Machine Learning Engineer Nanodegree

Deep Learning

Project: Build a Digit Recognition Program

In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.

Step 1: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize sequences of digits. Train the model using synthetic data generated by concatenating character images from notMNIST or MNIST. To produce a synthetic sequence of digits for testing, you can for example limit yourself to sequences up to five digits, and use five classifiers on top of your deep network. You would have to incorporate an additional ‘blank’ character to account for shorter number sequences.

There are various aspects to consider when thinking about this problem:

  • Your model can be derived from a deep neural net or a convolutional network.
  • You could experiment sharing or not the weights between the softmax classifiers.
  • You can also use a recurrent network in your deep neural net to replace the classification layers and directly emit the sequence of digits one-at-a-time.

Here is an example of a published baseline model on this problem. (video)

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [1]:
from __future__ import print_function

from keras.datasets import mnist
from keras.models import Model
from keras.layers import Input, Dense, TimeDistributed
from keras.layers import LSTM
from keras.utils import np_utils

# Training parameters.
batch_size = 32
nb_classes = 10
nb_epochs = 5

# Embedding dimensions.
row_hidden = 128
col_hidden = 128

# The data, shuffled and split between train and test sets.
(X_train, y_train), (X_test, y_test) = mnist.load_data()
Using TensorFlow backend.
In [2]:
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
In [3]:
import numpy as np
import pandas as pd
from time import time
from IPython.display import display 
from __future__ import print_function
import matplotlib.pyplot as plt
import numpy as np
import os
import sys
import tarfile
%matplotlib inline
import random
import math
from IPython.display import Image
from scipy import ndimage
from six.moves.urllib.request import urlretrieve
from six.moves import cPickle as pickle
blank=np.zeros((28,28),dtype=np.int)
fig=plt.figure()
plt.imshow(blank)
plt.show()
In [5]:
int1=list()
import random
for i in range(1,60001):
 int1.append(random.randint(1,59999))
print(len(int1))


int2=list()
import random
for i in range(1,60001):
 int2.append(random.randint(1,59999))
print(len(int1))


int3=list()
import random
for i in range(1,60001):
 int3.append(random.randint(1,59999))
print(len(int1))
60000
60000
60000
In [6]:
int3=list()
import random
for i in range(1,60001):
 int3.append(random.randint(1,59999))
print(len(int1))
60000
In [7]:
k1=list()
import random
for i in range(1,10001):
 k1.append(random.randint(1,60000))
print(len(k1))

k2=list()
import random
for i in range(1,10001):
 k2.append(random.randint(1,60000))
print(len(k2))

k3=list()
import random
for i in range(1,10001):
 k3.append(random.randint(1,60000))
print(len(k3))
10000
10000
10000
In [8]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = X_train.shape[0]
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for numbers in k:
      if i in k1:
        
        temp = np.hstack([blank])     
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str( '10'))
        
       
        data_labels.append(temp_str)
        
        
        w +=1
        i += 1
        np.array(data_labels)
      else:
       #if i in int1:
        temp = np.hstack([X_train[i]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(y_train[i]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset1, data_labels_train1 = createSequences()
from keras.utils import np_utils

print(dataset1.shape)

print(len(data_labels_train1))
print(data_labels_train1[1])
(60000, 28, 28)
60000
0
In [9]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = X_train.shape[0]
#dataset_size = 90000
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     for j in int2:
        j=random.randint(1,59999)
        temp = np.hstack([dataset1[j]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(data_labels_train1[j]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset2_temp, data_labels_train2_temp = createSequences()
from keras.utils import np_utils

print(dataset2_temp.shape)

print(len(data_labels_train2_temp))
print(data_labels_train2_temp[1])
(60000, 28, 28)
60000
0
In [10]:
X_train1=dataset2_temp
Y_train1=data_labels_train2_temp
print (X_train1.shape)
(60000, 28, 28)
In [11]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = X_train.shape[0]
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for numbers in k:
      if i in k2:
        
        temp = np.hstack([blank])     
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str( '10'))
        
       
        data_labels.append(temp_str)
        
        
        w +=1
        i += 1
        np.array(data_labels)
      else:
       #if i in int1:
        temp = np.hstack([dataset2_temp[i]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(data_labels_train2_temp[i]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset2, data_labels_train2 = createSequences()
from keras.utils import np_utils

print(dataset2.shape)

print(len(data_labels_train2))
print(data_labels_train2[1])
(60000, 28, 28)
60000
0
In [12]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = X_train.shape[0]
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for j in int3:
        j=random.randint(1,59999)
        temp = np.hstack([dataset2[j]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(data_labels_train2[j]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset3_temp, data_labels_train3_temp = createSequences()
from keras.utils import np_utils

print(dataset3_temp.shape)

print(len(data_labels_train3_temp))
print(data_labels_train3_temp[1])
(60000, 28, 28)
60000
0
In [13]:
X_train2=dataset3_temp
Y_train2=data_labels_train3_temp
In [14]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = X_train.shape[0]
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for numbers in k:
      if i in k2:
        
        temp = np.hstack([blank])     
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str( '10'))
        
       
        data_labels.append(temp_str)
        
        
        w +=1
        i += 1
        np.array(data_labels)
      else:
       #if i in int1:
        temp = np.hstack([dataset2_temp[i]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(data_labels_train2_temp[i]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset3, data_labels_train3 = createSequences()
from keras.utils import np_utils

print(dataset2.shape)

print(len(data_labels_train3))
print(data_labels_train3[1])
(60000, 28, 28)
60000
0
In [15]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = X_train.shape[0]
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for j in int3:
        j=random.randint(1,59999)
        temp = np.hstack([dataset3[j]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(data_labels_train3[j]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset4_temp, data_labels_train4_temp = createSequences()
from keras.utils import np_utils

print(dataset4_temp.shape)

print(len(data_labels_train4_temp))
print(data_labels_train4_temp[1])
(60000, 28, 28)
60000
5
In [16]:
X_train3=dataset4_temp 
Y_train3=data_labels_train4_temp
print(X_train3.shape)
(60000, 28, 28)
In [17]:
X1=X_train1
X2=X_train2
X3=X_train3
In [19]:
dataset_size = X_train.shape[0]
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, 84),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     
        #temp = np.hstack([X_train1[i],X_train2[i],X_train3[i]])
        temp = np.hstack([X1[i],X2[i],X3[i]])
        
        dataset[i, :, :] = temp
        
        
        temp_str = (3,int(Y_train1[i]),int(Y_train2[i]),int(Y_train3[i]),10,10)
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return  dataset,data_labels

complete_dataset,Y_train_new = createSequences()
from keras.utils import np_utils



print(len(Y_train_new))
print(Y_train_new[1])
60000
(3, 0, 0, 5, 10, 10)
In [53]:
print(complete_dataset.shape)
(60000, 28, 84)
In [20]:
r1=list()
for i in range(1,4999):
 r1.append(random.randint(1,4999))
print(len(r1))
print (r1[100])

r2=list()
for i in range(1,4999):
 r2.append(random.randint(1,4999))
print(len(r2))
print (r2[100])

r3=list()
for i in range(1,4999):
 r3.append(random.randint(1,4999))
print(len(r3))
print (r3[100])
4998
4170
4998
4749
4998
3450
In [21]:
m1=list()
for i in range(1,500):
 m1.append(random.randint(1,4999))
print(len(m1))
print (m1[100])

m2=list()
for i in range(1,500):
 m2.append(random.randint(1,4999))
print(len(m2))
print (m2[100])

m3=list()
for i in range(1,500):
 m3.append(random.randint(1,4999))
print(len(m3))
print (m3[100])
499
1763
499
1884
499
4929
In [22]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size =10000
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for numbers in k:
      if i in m1:
        
        temp = np.hstack([blank])     
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str( '10'))
        
       
        data_labels.append(temp_str)
        
        
        w +=1
        i += 1
        np.array(data_labels)
      else:
       #if i in int1:
        temp = np.hstack([X_test[i]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(y_test[i]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset_test1, data_labels_test1 = createSequences()
from keras.utils import np_utils

print(dataset_test1.shape)

print(len(data_labels_test1))
print(data_labels_test1[1])
(10000, 28, 28)
10000
2
In [23]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = 10000
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels_test = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for j in int3:
        j=random.randint(1,9999)
        temp = np.hstack([dataset_test1[j]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(data_labels_test1[j]))
       
        
        data_labels_test.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels_test)

    return dataset, data_labels_test


X_test1, Y_test1 = createSequences()
from keras.utils import np_utils

print(X_test1.shape)

print(len(Y_test1))
print(Y_test1[1])
(10000, 28, 28)
10000
0
In [24]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size =10000
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for numbers in k:
      if i in m2:
        
        temp = np.hstack([blank])     
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str( '10'))
        
       
        data_labels.append(temp_str)
        
        
        w +=1
        i += 1
        np.array(data_labels)
      else:
       #if i in int1:
        temp = np.hstack([X_test[i]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(y_test[i]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset_test2, data_labels_test2 = createSequences()
from keras.utils import np_utils

print(dataset_test2.shape)

print(len(data_labels_test2))
print(data_labels_test2[1])
(10000, 28, 28)
10000
10
In [25]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = 10000
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels_test = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for j in int3:
        j=random.randint(1,9999)
        temp = np.hstack([dataset_test2[j]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(data_labels_test2[j]))
       
        
        data_labels_test.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels_test)

    return dataset, data_labels_test


X_test2, Y_test2 = createSequences()
from keras.utils import np_utils

print(X_test2.shape)

print(len(Y_test2))
print(Y_test2[1])
(10000, 28, 28)
10000
9
In [26]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size =10000
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for numbers in k:
      if i in m3:
        
        temp = np.hstack([blank])     
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str( '10'))
        
       
        data_labels.append(temp_str)
        
        
        w +=1
        i += 1
        np.array(data_labels)
      else:
       #if i in int1:
        temp = np.hstack([X_test[i]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(y_test[i]))
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels


dataset_test3, data_labels_test3 = createSequences()
from keras.utils import np_utils

print(dataset_test3.shape)

print(len(data_labels_test3))
print(data_labels_test3[1])
(10000, 28, 28)
10000
2
In [27]:
image_width = 28
image_height = 28
#dataset_size=X_train.shape[0]
dataset_size = 10000
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, image_width),dtype=np.float32)

    data_labels_test = []

    i = 0
    w = 0
    while i < (dataset_size):
     #for j in int3:
        j=random.randint(1,9999)
        temp = np.hstack([dataset_test3[j]])
        
        
        dataset[i, :, :] = temp
        
        
        temp_str = (str(data_labels_test3[j]))
       
        
        data_labels_test.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels_test)

    return dataset, data_labels_test


X_test3, Y_test3= createSequences()
from keras.utils import np_utils

print(X_test3.shape)

print(len(Y_test3))
print(Y_test3[1])
(10000, 28, 28)
10000
0
In [28]:
XX=X_test1
XY=X_test2
XZ=X_test3
In [29]:
print(X_test1.shape) 
num_pixels = X_test1.shape[1]*X_test1.shape[2]
X_test1 = X_test1.reshape(X_test1.shape[0], num_pixels).astype('float32')
X_test2 = X_test2.reshape(X_test2.shape[0], num_pixels).astype('float32')
X_test3 = X_test3.reshape(X_test3.shape[0], num_pixels).astype('float32')
X_test1 /= 255
X_test2 /= 255
X_test3 /= 255
#X_test /= 255
print(X_test1.shape)
(10000, 28, 28)
(10000, 784)
In [30]:
dataset_size =10000
def createSequences():
    dataset = np.ndarray(shape=(dataset_size, image_height, 84),dtype=np.float32)

    data_labels = []

    i = 0
    w = 0
    while i < (dataset_size):
     
        #temp = np.hstack([X_train1[i]])
        
        
        #dataset[i, :, :] = temp
        
        temp = np.hstack([XX[i],XY[i],XZ[i]])
        
        dataset[i, :, :] = temp
        temp_str = (3,int(Y_test1[i]),int(Y_test2[i]),int(Y_test3[i]),10,10)
       
        
        data_labels.append(temp_str)
        
        
        w += 1
        i += 1
        np.array(data_labels)

    return dataset, data_labels

test_dataset,Y_test_new = createSequences()
from keras.utils import np_utils



print(len(Y_test_new))
print(Y_test_new[1])
10000
(3, 0, 9, 0, 10, 10)
In [31]:
X_train = complete_dataset.reshape(complete_dataset.shape[0], 2352).astype('float32')
X_test = test_dataset.reshape(test_dataset.shape[0], 2352).astype('float32')
In [32]:
-
60000
3
In [39]:
dataset_size =10000
def createSequences():
    #dataset = np.ndarray(shape=(dataset_size, image_height, 84),dtype=np.float32)

    data_labels1 = []
    data_labels2 = []
    data_labels3 = []
    data_labels4 = []
    data_labels5 = []
    data_labels6 = []

    i = 0
    w = 0
    while i < (dataset_size):
     
        #temp = np.hstack([X_train1[i]])
        
        
        #dataset[i, :, :] = temp
        
        #temp = np.hstack([XX[i],XY[i],XZ[i]])
        
        #dataset[i, :, :] = temp
        temp_str1 = (Y_test_new[i][0])
        temp_str2 = (Y_test_new[i][1])
        temp_str3 = (Y_test_new[i][2])
        temp_str4 = (Y_test_new[i][3])
        temp_str5 = (Y_test_new[i][4])
        temp_str6 = (Y_test_new[i][5])
       
        
        data_labels1.append(temp_str1)
        data_labels2.append(temp_str2)
        data_labels3.append(temp_str3)
        data_labels4.append(temp_str4)
        data_labels5.append(temp_str5)
        data_labels6.append(temp_str6)
        
        
        w += 1
        i += 1
        np.array(data_labels1)
        np.array(data_labels2)
        np.array(data_labels3)
        np.array(data_labels4)
        np.array(data_labels5)
        np.array(data_labels6)

    return data_labels1,data_labels2,data_labels3,data_labels4,data_labels5,data_labels6

y_1,y_2,y_3,y_4,y_5,y_6 = createSequences()
from keras.utils import np_utils



print(len(y_1))
print(y_1[1])
10000
3
In [34]:
Y1 = np_utils.to_categorical(y1, 11)
Y2 = np_utils.to_categorical(y2, 11)
Y3 = np_utils.to_categorical(y3, 11)
Y4 = np_utils.to_categorical(y4, 11)
Y5 = np_utils.to_categorical(y5, 11)
Y6 = np_utils.to_categorical(y6, 11)
In [40]:
Y_1 = np_utils.to_categorical(y_1, 11)
Y_2 = np_utils.to_categorical(y_2, 11)
Y_3 = np_utils.to_categorical(y_3, 11)
Y_4 = np_utils.to_categorical(y_4, 11)
Y_5 = np_utils.to_categorical(y_5, 11)
Y_6 = np_utils.to_categorical(y_6, 11)
In [33]:
print(len(Y_train_new))
60000
In [35]:
print(X_train.shape)
x=X_train
(60000, 2352)
In [50]:
print(test_dataset.shape)
fig=plt.figure()
plt.imshow(test_dataset[1])
plt.show()
(10000, 28, 84)
In [36]:
X_train1 = complete_dataset.reshape(complete_dataset.shape[0], 28,84,1).astype('float32')
In [41]:
X_test1 = test_dataset.reshape(test_dataset.shape[0], 28,84,1).astype('float32')
In [37]:
from keras.layers import Input, Dense
from keras.models import Model
from keras.layers import Dropout
from keras.layers import Flatten
from keras.layers.convolutional import Convolution2D
from keras.layers.convolutional import MaxPooling2D


inputs = Input(shape=(28,84,1))
conv = Convolution2D(32, 5, 5, border_mode='same', input_shape=( 28,84,1), activation='relu')
conv1 = Convolution2D(64, 5, 5, border_mode='same',  activation='relu')
conv2= Convolution2D(128, 5, 5, border_mode='same',  activation='relu')
x=conv(inputs)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv1(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv2(x)
x=MaxPooling2D(pool_size=(2, 2))(x)

x=Flatten()(x)
x=Dropout(0.2)(x)
x = Dense(1064, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(800, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(600, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(400, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(200, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(164, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(32, activation='relu')(x)
predictions1 = Dense(11, activation='softmax')(x)
predictions2 = Dense(11, activation='softmax')(x)
predictions3 = Dense(11, activation='softmax')(x)
predictions4 = Dense(11, activation='softmax')(x)
predictions5 = Dense(11, activation='softmax')(x)
predictions6= Dense(11, activation='softmax')(x)

model = Model(input=inputs, output=[predictions1,predictions2,predictions3,predictions4,predictions5,predictions6])
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
In [107]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
60000/60000 [==============================] - 421s - loss: 1.9934 - dense_58_loss: 0.0626 - dense_59_loss: 0.6333 - dense_60_loss: 0.5659 - dense_61_loss: 0.6057 - dense_62_loss: 0.0434 - dense_63_loss: 0.0825 - dense_58_acc: 0.9975 - dense_59_acc: 0.8108 - dense_60_acc: 0.8359 - dense_61_acc: 0.8213 - dense_62_acc: 0.9984 - dense_63_acc: 0.9862   
Epoch 2/10
60000/60000 [==============================] - 426s - loss: 0.3901 - dense_58_loss: 1.1953e-07 - dense_59_loss: 0.1403 - dense_60_loss: 0.1280 - dense_61_loss: 0.1219 - dense_62_loss: 1.1923e-07 - dense_63_loss: 1.1948e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.9686 - dense_60_acc: 0.9702 - dense_61_acc: 0.9721 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Epoch 3/10
60000/60000 [==============================] - 446s - loss: 0.3793 - dense_58_loss: 1.2942e-07 - dense_59_loss: 0.1404 - dense_60_loss: 0.1232 - dense_61_loss: 0.1157 - dense_62_loss: 1.3675e-07 - dense_63_loss: 1.8544e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.9726 - dense_60_acc: 0.9744 - dense_61_acc: 0.9774 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Epoch 4/10
60000/60000 [==============================] - 444s - loss: 0.5162 - dense_58_loss: 1.1945e-07 - dense_59_loss: 0.1747 - dense_60_loss: 0.1805 - dense_61_loss: 0.1610 - dense_62_loss: 1.1925e-07 - dense_63_loss: 1.2059e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.9723 - dense_60_acc: 0.9728 - dense_61_acc: 0.9754 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Epoch 5/10
60000/60000 [==============================] - 443s - loss: 0.4176 - dense_58_loss: 1.1921e-07 - dense_59_loss: 0.1591 - dense_60_loss: 0.1323 - dense_61_loss: 0.1263 - dense_62_loss: 1.1921e-07 - dense_63_loss: 1.1921e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.9735 - dense_60_acc: 0.9777 - dense_61_acc: 0.9782 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Epoch 6/10
60000/60000 [==============================] - 444s - loss: 0.7398 - dense_58_loss: 1.1921e-07 - dense_59_loss: 0.2313 - dense_60_loss: 0.2718 - dense_61_loss: 0.2367 - dense_62_loss: 1.1921e-07 - dense_63_loss: 1.1921e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.9711 - dense_60_acc: 0.9691 - dense_61_acc: 0.9728 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Epoch 7/10
60000/60000 [==============================] - 444s - loss: 0.5208 - dense_58_loss: 1.1921e-07 - dense_59_loss: 0.1935 - dense_60_loss: 0.1751 - dense_61_loss: 0.1522 - dense_62_loss: 1.1921e-07 - dense_63_loss: 1.1921e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.9731 - dense_60_acc: 0.9756 - dense_61_acc: 0.9780 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Epoch 8/10
60000/60000 [==============================] - 445s - loss: 0.5786 - dense_58_loss: 1.1921e-07 - dense_59_loss: 0.2133 - dense_60_loss: 0.2051 - dense_61_loss: 0.1602 - dense_62_loss: 1.1921e-07 - dense_63_loss: 1.1921e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.9721 - dense_60_acc: 0.9737 - dense_61_acc: 0.9772 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Epoch 9/10
60000/60000 [==============================] - 445s - loss: 1.8310 - dense_58_loss: 1.1921e-07 - dense_59_loss: 0.7761 - dense_60_loss: 0.4898 - dense_61_loss: 0.5651 - dense_62_loss: 1.1921e-07 - dense_63_loss: 1.1921e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.9372 - dense_60_acc: 0.9558 - dense_61_acc: 0.9525 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Epoch 10/10
60000/60000 [==============================] - 442s - loss: 4.8229 - dense_58_loss: 1.1921e-07 - dense_59_loss: 1.6949 - dense_60_loss: 1.5408 - dense_61_loss: 1.5872 - dense_62_loss: 1.1921e-07 - dense_63_loss: 1.1921e-07 - dense_58_acc: 1.0000 - dense_59_acc: 0.8826 - dense_60_acc: 0.8935 - dense_61_acc: 0.8906 - dense_62_acc: 1.0000 - dense_63_acc: 1.0000   
Out[107]:
<keras.callbacks.History at 0x7fdf81075d10>
In [38]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
60000/60000 [==============================] - 445s - loss: 3.1074 - dense_21_loss: 0.0253 - dense_22_loss: 0.9979 - dense_23_loss: 1.0301 - dense_24_loss: 1.0129 - dense_25_loss: 0.0237 - dense_26_loss: 0.0175 - dense_21_acc: 0.9990 - dense_22_acc: 0.6814 - dense_23_acc: 0.6692 - dense_24_acc: 0.6709 - dense_25_acc: 0.9994 - dense_26_acc: 0.9996         
Epoch 2/10
60000/60000 [==============================] - 479s - loss: 1.0311 - dense_21_loss: 1.1968e-07 - dense_22_loss: 0.3508 - dense_23_loss: 0.3429 - dense_24_loss: 0.3373 - dense_25_loss: 1.1977e-07 - dense_26_loss: 1.1944e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.8995 - dense_23_acc: 0.9040 - dense_24_acc: 0.9011 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Epoch 3/10
60000/60000 [==============================] - 484s - loss: 0.7981 - dense_21_loss: 4.9656e-07 - dense_22_loss: 0.2733 - dense_23_loss: 0.2669 - dense_24_loss: 0.2578 - dense_25_loss: 5.3141e-07 - dense_26_loss: 3.9030e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.9309 - dense_23_acc: 0.9320 - dense_24_acc: 0.9329 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Epoch 4/10
60000/60000 [==============================] - 494s - loss: 0.7390 - dense_21_loss: 1.1961e-07 - dense_22_loss: 0.2554 - dense_23_loss: 0.2440 - dense_24_loss: 0.2396 - dense_25_loss: 1.1951e-07 - dense_26_loss: 1.1977e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.9401 - dense_23_acc: 0.9411 - dense_24_acc: 0.9414 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Epoch 5/10
60000/60000 [==============================] - 495s - loss: 0.7397 - dense_21_loss: 1.1973e-07 - dense_22_loss: 0.2545 - dense_23_loss: 0.2466 - dense_24_loss: 0.2386 - dense_25_loss: 1.1972e-07 - dense_26_loss: 1.1944e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.9441 - dense_23_acc: 0.9446 - dense_24_acc: 0.9462 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Epoch 6/10
60000/60000 [==============================] - 496s - loss: 0.7316 - dense_21_loss: 1.1935e-07 - dense_22_loss: 0.2553 - dense_23_loss: 0.2453 - dense_24_loss: 0.2311 - dense_25_loss: 1.1934e-07 - dense_26_loss: 1.1935e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.9467 - dense_23_acc: 0.9473 - dense_24_acc: 0.9492 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Epoch 7/10
60000/60000 [==============================] - 492s - loss: 0.7577 - dense_21_loss: 1.2112e-07 - dense_22_loss: 0.2641 - dense_23_loss: 0.2539 - dense_24_loss: 0.2397 - dense_25_loss: 1.2096e-07 - dense_26_loss: 1.2153e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.9464 - dense_23_acc: 0.9463 - dense_24_acc: 0.9498 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Epoch 8/10
60000/60000 [==============================] - 496s - loss: 0.8877 - dense_21_loss: 1.3673e-07 - dense_22_loss: 0.3068 - dense_23_loss: 0.2880 - dense_24_loss: 0.2928 - dense_25_loss: 1.4551e-07 - dense_26_loss: 1.3409e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.9435 - dense_23_acc: 0.9446 - dense_24_acc: 0.9462 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Epoch 9/10
60000/60000 [==============================] - 490s - loss: 1.4231 - dense_21_loss: 1.2027e-07 - dense_22_loss: 0.4782 - dense_23_loss: 0.4765 - dense_24_loss: 0.4684 - dense_25_loss: 1.1998e-07 - dense_26_loss: 1.1984e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.9334 - dense_23_acc: 0.9344 - dense_24_acc: 0.9362 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Epoch 10/10
60000/60000 [==============================] - 495s - loss: 0.8226 - dense_21_loss: 1.4819e-07 - dense_22_loss: 0.2759 - dense_23_loss: 0.2831 - dense_24_loss: 0.2635 - dense_25_loss: 1.8333e-07 - dense_26_loss: 1.4210e-07 - dense_21_acc: 1.0000 - dense_22_acc: 0.9470 - dense_23_acc: 0.9459 - dense_24_acc: 0.9496 - dense_25_acc: 1.0000 - dense_26_acc: 1.0000     
Out[38]:
<keras.callbacks.History at 0x7fe983f22e50>
In [42]:
print('started............')
scores = model.evaluate(X_test1, [Y_1,Y_2,Y_3,Y_4,Y_5,Y_6], verbose=0)
print('Test loss and Test Accuracy', scores)
started............
Test loss and Test Accuracy [0.45342636806145309, 1.1920928955078125e-07, 0.1266812288660556, 0.1714570570329437, 0.15528772211412434, 1.1920928955078125e-07, 1.1920928955078125e-07, 1.0, 0.98109999999999997, 0.97609999999999997, 0.9778, 1.0, 1.0]

from keras.datasets import mnist

from keras.models import Sequential from keras.layers import Dense from keras.layers import Dropout from keras.layers import Flatten from keras.layers.convolutional import Convolution2D from keras.layers.convolutional import MaxPooling2D from keras.utils import np_utils

model = baseline_model()

Fit the model

model.fit(X_train, y_train, validation_data=(X_test, y_test), nb_epoch=10, batch_size=200, verbose=2)

Final evaluation of the model

Question 1

What approach did you take in coming up with a solution to this problem?

Handwritten digit recognition is a typical image classification problem. Convolutional neural networks (convnets) have shown excellent results in various image classification tasks. Many researchers in used convnetsto solve image classification problems. Md.Shopon et al in “Bangla handwritten digit recognition using autoencoder and deep convolutional neural network “ used convnets for Bangla handwritten digit recognition. Chuankun Li et al in “Joint Distance Maps Based Action Recognition with Convolutional Neural Network” used convnets to encode the spatio-temporal information of skeleton sequences into color texture images, referred to as Joint Distance Maps (JDMs). Van Tuan Nguyen in “ConvNets and AGMM based real-time human detection under fisheye camera for embedded surveillance” proposed ConvNets based YOLO model for a successful object (including human) detection. Pichao Wang et al in “Action Recognition From Depth Maps Using Deep Convolutional Neural Networks” proposed convnets for human action recognition.Vladimir Risojevic et al in “Analysis of learned features for remote sensing image classification” used convnets for analysis of learned features for remote sensing image classification.

The model is derived using Convolutional Networks(convnets), I used three classifiers with convnets to identify digits upto three.

Question 2

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.)

The model is designed using Keras API

The model consists of 14 layer convnets,

Layer1: convolutional layer with convolution size: 5 5 32

Layer2:Maxpooling layer of pool size 2*2

Layer3:convolutional layer with convolution size: 5 5 64

Layer4:Maxpooling layer of pool size 2*2

Layer5:convolutional layer with convolution size: 5 5 128

Layer6:Maxpooling layer of pool size 2*2

Dropout of 0.2

Layer7:Fully Connected Layer with 1064 outputs

Dropout of 0.2

Layer8:Fully Connected Layer with 800 outputs

Dropout of 0.2

Layer9:Fully Connected Layer withf 164 outputs

Dropout of 0.2

Layer13:Fully Connected Layer with 32 outputs

Dropout of 0.2

Layer14: Output layer with 11 outputs

In our Convolutional Neural Networks (CNN), we have used 3 Convolutional (CNV) layers followed by 3 Max-Pooling (PL) layers which is followed by 5 Fully Connected (FC) layers as given above.

We receive 28x84 inputs from synthetic dataset. We have used 5x5x32 filters at the first CNV layers ,5x5x64 filters in the second CNV layers and 5x5x128 filters in the third CNV layers

In the Pooling Layers, we apply max(0,x) (Rectifier Linear Unit) after each CNV which is also followed by 2x2 maximum subsampling (For every 2x2 window, maximum value within the window is passed to next layer).

At the input of fully connected layer, we have 1064 samples. In first FC layer we reduce the size to 800 samples, in the second 164 , in the third 32 and the last FC reduce the size to 11 samples which is the number of classification categories we have in synthetic datasets. The Layer14 is a simple SoftMax Classifier

Question 3

How did you train your model? How did you generate your synthetic dataset? Include examples of images from the synthetic data you constructed.

I trained the model using three digit synthetic dataset prepared from mnist training dataset.

GENERATION OF SYNTHETIC DATASET

  1. I created a blank image of size 28*28
  2. I mixed the blank image randomly to the training set of mnist daytaset and shuffled it randomly and called X_train1
  3. I mixed the blank image randomly to the training set of mnist daytaset and shuffled it randomly and called X_train2
  4. I mixed the blank image randomly to the training set of mnist daytaset and shuffled it randomly and called X_train3
  5. So I prepared three different datasets X_train1, X_train2 and X_train3
  6. These three datatsets consists of image of one digit and of size 28*28
  7. I mixed these three datatsets from their 0th image to 60000 image to prepare three digit image (i.e for examle 0th images X_train1[0]+X_train2[0]+X_train3[0]) of size 28*84
  8. AboveI provided exaple of three digit dataset I prepared
  9. I trained the dataset using convnet model explained above.
  10. I trained 60,000images which form train dataset(X_train1)
  11. There are six target variable Y1,Y2,Y3,Y4,Y5,Y6
  12. I used 20 epochs with batch size 32

Step 2: Train a Model on a Realistic Dataset

Once you have settled on a good architecture, you can train your model on real data. In particular, the Street View House Numbers (SVHN) dataset is a good large-scale dataset collected from house numbers in Google Street View. Training on this more challenging dataset, where the digits are not neatly lined-up and have various skews, fonts and colors, likely means you have to do some hyperparameter exploration to perform well.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [8]:
from __future__ import print_function
import matplotlib.pyplot as plt
import numpy as np
import os
import sys
import tarfile
import tensorflow as tf
from IPython.display import Image
from scipy import ndimage
from six.moves.urllib.request import urlretrieve
from six.moves import cPickle as pickle
%matplotlib inline
In [2]:
url = 'http://ufldl.stanford.edu/housenumbers/'
last_percent_reported = None


def download_progress_hook(count, blockSize, totalSize):
    """
    A hook to report the progress of a download. This is mostly intended for users with
    slow internet connections. Reports every 1% change in download progress.
    """
    global last_percent_reported
    percent = int(count * blockSize * 100 / totalSize)

    if last_percent_reported != percent:
        if percent % 5 == 0:
            sys.stdout.write("%s%%" % percent)
            sys.stdout.flush()
        else:
            sys.stdout.write(".")
            sys.stdout.flush()

        last_percent_reported = percent
        
        
def maybe_download(filename, force=False):
    """
    Download a file if not present, and make sure it's the right size.
    """
    if force or not os.path.exists(filename):
        print('Attempting to download:', filename) 
        filename, _ = urlretrieve(url + filename, filename, reporthook=download_progress_hook)
        print('\nDownload Complete!')
    else:
        print(filename, 'is already downloaded. Skipped.')
    return filename
In [3]:
train_filename = maybe_download('train.tar.gz')
train.tar.gz is already downloaded. Skipped.
In [4]:
test_filename = maybe_download('test.tar.gz')
test.tar.gz is already downloaded. Skipped.
In [5]:
extra_filename = maybe_download('extra.tar.gz')
extra.tar.gz is already downloaded. Skipped.
In [6]:
np.random.seed(133)


def maybe_extract(file_, force=False):
    filename = os.path.splitext(os.path.splitext(file_)[0])[0]  # remove .tar.gz
    
    if os.path.isdir(filename) and not force:
        # You may override by setting force=True.
        print('%s is already presented - Skipping extraction of %s.' % (filename, file_))
    else:
        print('Extracting %s file data. Please wait...' % file_)
        tar = tarfile.open(file_)
        sys.stdout.flush()
        tar.extractall()
        tar.close()
        print('File %s is successfully extracted into %s directory.' % (file_, filename))        
    
    return filename
In [7]:
train_folder = maybe_extract(train_filename)
Extracting train.tar.gz file data. Please wait...
File train.tar.gz is successfully extracted into train directory.
In [8]:
test_folder = maybe_extract(test_filename)
Extracting test.tar.gz file data. Please wait...
File test.tar.gz is successfully extracted into test directory.
In [9]:
extra_folder = maybe_extract(extra_filename)
Extracting extra.tar.gz file data. Please wait...
File extra.tar.gz is successfully extracted into extra directory.
In [12]:
def remove_anomaly_samples(data, max_class_length = 5):
    """
    Here we remove all data which has class length higher than specified value.
    """
    print("\nDataset size before update:", len(data))
    
    for i in range(len(data)):
        if i < len(data) and len(data[i]['label']) > max_class_length:
            print("\nAnomaly at index %d detected. Class size: %d" % (i, len(data[i]['label'])))
            del data[i]
            
    print("\nDataset after before update:", len(data))            
    return data
In [13]:
import h5py


class DigitStructsWrapper:
    def __init__(self, file_, start_ = 0, end_ = 0):
        self.file_ = h5py.File(file_, 'r')
        self.names = self.file_['digitStruct']['name'][start_:end_] if end_ > 0 else self.file_['digitStruct']['name']
        self.bboxes = self.file_['digitStruct']['bbox'][start_:end_] if end_ > 0 else self.file_['digitStruct']['bbox']
        self.collectionSize = len(self.names)
        print("\n%s file structure contain %d entries" % (file_, self.collectionSize))
        
        
    def bboxHelper(self, keys_):
        """
        Method handles the coding difference when there is exactly one bbox or an array of bbox. 
        """
        if (len(keys_) > 1):
            val = [self.file_[keys_.value[j].item()].value[0][0] for j in range(len(keys_))]
        else:
            val = [keys_.value[0][0]]
        return val

    
    # getBbox returns a dict of data for the n(th) bbox. 
    def getBbox(self, n):
        bbox = {}
        bb = self.bboxes[n].item()
        bbox['height'] = self.bboxHelper(self.file_[bb]["height"])
        bbox['left'] = self.bboxHelper(self.file_[bb]["left"])
        bbox['top'] = self.bboxHelper(self.file_[bb]["top"])
        bbox['width'] = self.bboxHelper(self.file_[bb]["width"])
        bbox['label'] = self.bboxHelper(self.file_[bb]["label"])
        return bbox

    
    def getName(self, n):
        """
        Method returns the filename for the n(th) digitStruct. Since each letter is stored in a structure 
        as array of ANSII char numbers we should convert it back by calling chr function.
        """
        return ''.join([chr(c[0]) for c in self.file_[self.names[n][0]].value])

    
    def getNumberStructure(self,n):
        s = self.getBbox(n)
        s['name']=self.getName(n)
        return s

    def getAllNumbersStructure(self):
        """
        Method returns an array, which contains information about every image.
        This info contains: positions, labels 
        """
        return [self.getNumberStructure(i) for i in range(self.collectionSize)]

    
    
    def getAllNumbersRestructured(self): 
        numbersData = self.getAllNumbersStructure()
        print("\nObject structure before transforming: ", numbersData[0])
        remove_anomaly_samples(numbersData)
        
        result = []
        for numData in numbersData:
            metadatas = []
            for i in range(len(numData['height'])):
                metadata = {}
                metadata['height'] = numData['height'][i]
                metadata['label']  = numData['label'][i]
                metadata['left']   = numData['left'][i]
                metadata['top']    = numData['top'][i]
                metadata['width']  = numData['width'][i]
                metadatas.append(metadata)
                
            result.append({ 'boxes':metadatas, 'name':numData["name"] })
            
        print("\nObject structure after transforming: ", result[0])
        
        return result
In [14]:
file_ = os.path.join(train_folder, 'digitStruct.mat')
dsf = DigitStructsWrapper(file_)
train_data = dsf.getAllNumbersRestructured()
train/digitStruct.mat file structure contain 33402 entries

Object structure before transforming:  {'name': '1.png', 'top': [77.0, 81.0], 'label': [1.0, 9.0], 'width': [81.0, 96.0], 'height': [219.0, 219.0], 'left': [246.0, 323.0]}

Dataset size before update: 33402

Anomaly at index 29929 detected. Class size: 6

Dataset after before update: 33401

Object structure after transforming:  {'boxes': [{'width': 81.0, 'top': 77.0, 'label': 1.0, 'left': 246.0, 'height': 219.0}, {'width': 96.0, 'top': 81.0, 'label': 9.0, 'left': 323.0, 'height': 219.0}], 'name': '1.png'}
In [15]:
file_ = os.path.join(test_folder, 'digitStruct.mat')
dsf = DigitStructsWrapper(file_)
test_data = dsf.getAllNumbersRestructured()
test/digitStruct.mat file structure contain 13068 entries

Object structure before transforming:  {'name': '1.png', 'top': [7.0], 'label': [5.0], 'width': [19.0], 'height': [30.0], 'left': [43.0]}

Dataset size before update: 13068

Dataset after before update: 13068

Object structure after transforming:  {'boxes': [{'width': 19.0, 'top': 7.0, 'label': 5.0, 'left': 43.0, 'height': 30.0}], 'name': '1.png'}
In [16]:
file_ = os.path.join(extra_folder, 'digitStruct.mat')
dsf = DigitStructsWrapper(file_, 0, 50000)
extra_data = dsf.getAllNumbersRestructured()
extra/digitStruct.mat file structure contain 50000 entries

Object structure before transforming:  {'name': '1.png', 'top': [70.0, 41.0, 23.0], 'label': [4.0, 7.0, 8.0], 'width': [38.0, 36.0, 47.0], 'height': [56.0, 56.0, 56.0], 'left': [24.0, 55.0, 79.0]}

Dataset size before update: 50000

Dataset after before update: 50000

Object structure after transforming:  {'boxes': [{'width': 38.0, 'top': 70.0, 'label': 4.0, 'left': 24.0, 'height': 56.0}, {'width': 36.0, 'top': 41.0, 'label': 7.0, 'left': 55.0, 'height': 56.0}, {'width': 47.0, 'top': 23.0, 'label': 8.0, 'left': 79.0, 'height': 56.0}], 'name': '1.png'}
In [17]:
from PIL import Image

def print_data_stats(data, folder):
    data_imgSize = np.ndarray([len(data),2])

    for i in np.arange(len(data)):
        filename = data[i]['name']
        filepath = os.path.join(folder, filename)
        data_imgSize[i, :] = Image.open(filepath).size[:]

    max_w, max_h = np.amax(data_imgSize[:,0]), np.amax(data_imgSize[:,1])
    min_w, min_h = np.amin(data_imgSize[:,0]), np.amin(data_imgSize[:,1])
    mean_w, mean_h = np.mean(data_imgSize[:,0]), np.mean(data_imgSize[:,1])
    print(folder, "max width and height:", max_w, max_h) 
    print(folder, "min width and height:", min_w, min_h)
    print(folder, "mean width and height:", mean_w, mean_h, "\n")
    
    max_w_i, max_h_i = np.where(data_imgSize[:,0] == max_w), np.where(data_imgSize[:,1] == max_h)
    print(folder, "max width indicies:", max_w_i) 
    print(folder, "max height indicies:", max_h_i, "\n")
    
    
    min_w_i, min_h_i = np.where(data_imgSize[:,0] == min_w), np.where(data_imgSize[:,1] == min_h)
    print(folder, "min width indicies:", min_w_i) 
    print(folder, "min height indicies:", min_h_i, "\n***\n")
In [18]:
print_data_stats(train_data, train_folder)
print_data_stats(test_data, test_folder)
print_data_stats(extra_data, extra_folder)
train max width and height: 876.0 501.0
train min width and height: 25.0 12.0
train mean width and height: 128.286338732 57.2139456902 

train max width indicies: (array([  410,  4163, 15855, 30483]),)
train max height indicies: (array([15855]),) 

train min width indicies: (array([9747]),)
train min height indicies: (array([ 1813,  2291,  4829,  5691,  9488,  9747,  9831, 10175, 10938,
       14902, 16284, 20314, 20775, 21544, 22330, 24015, 25438, 26047,
       26345, 27062, 27160, 27593, 27959, 29526, 29701, 30064, 30089,
       30462, 30947, 32339, 32351, 32539, 32567, 33141, 33180, 33202]),) 
***

test max width and height: 1083.0 516.0
test min width and height: 31.0 13.0
test mean width and height: 172.583486379 71.5664983165 

test max width indicies: (array([ 1722,  2949,  6233, 12862]),)
test max height indicies: (array([14]),) 

test min width indicies: (array([  459,  5352,  7776, 11257, 12191]),)
test min height indicies: (array([ 145, 1591, 5352, 7776]),) 
***

extra max width and height: 668.0 356.0
extra min width and height: 22.0 13.0
extra mean width and height: 100.04358 60.72528 

extra max width indicies: (array([32352]),)
extra max height indicies: (array([43845]),) 

extra min width indicies: (array([19731, 25534]),)
extra min height indicies: (array([ 9662, 26875, 27476, 42560, 49492]),) 
***

In [19]:
img_size = 32

def prepare_images(samples, folder):
    print("Started preparing images for convnet...")
    
    prepared_images = np.ndarray([len(samples),img_size,img_size,1], dtype='float32')
    actual_numbers = np.ones([len(samples),6], dtype=int) * 10
    files = []
    for i in range(len(samples)):
        filename = samples[i]['name']
        filepath = os.path.join(folder, filename)
        image = Image.open(filepath)
        boxes = samples[i]['boxes']
        number_length = len(boxes)
        files.append(filename)
        
        # at 0 index we store length of a label. 3 -> 1; 123-> 3, 12543 -> 5
        actual_numbers[i,0] = number_length
        
        top = np.ndarray([number_length], dtype='float32')
        left = np.ndarray([number_length], dtype='float32')
        height = np.ndarray([number_length], dtype='float32')
        width = np.ndarray([number_length], dtype='float32')
        
        for j in range(number_length):
            
            actual_numbers[i,j+1] = boxes[j]['label']
            if boxes[j]['label'] == 10: 
                actual_numbers[i,j+1] = 0
                
            top[j] = boxes[j]['top']
            left[j] = boxes[j]['left']
            height[j] = boxes[j]['height']
            width[j] = boxes[j]['width']
        
        img_min_top = np.amin(top)
        img_min_left = np.amin(left)
        img_height = np.amax(top) + height[np.argmax(top)] - img_min_top
        img_width = np.amax(left) + width[np.argmax(left)] - img_min_left

        img_left = np.floor(img_min_left - 0.1 * img_width)
        img_top = np.floor(img_min_top - 0.1 * img_height)
        img_right = np.amin([np.ceil(img_left + 1.2 * img_width), image.size[0]])
        img_bottom = np.amin([np.ceil(img_top + 1.2 * img_height), image.size[1]])
            
        image = image.crop((int(img_left), int(img_top), int(img_right), int(img_bottom))).resize([img_size, img_size], Image.ANTIALIAS) # Resize image to 32x32
        image = np.dot(np.array(image, dtype='float32'), [[0.2989],[0.5870],[0.1140]]) # Convert image to the grayscale

        mean = np.mean(image, dtype='float32')
        std = np.std(image, dtype='float32', ddof=1)
        if std < 0.0001: 
            std = 1.0
        image = (image - mean) / std
        prepared_images[i,:,:] = image[:,:,:]
        
    print("Completed. Images cropped, resized and grayscaled")
    
    return prepared_images, actual_numbers, files
In [20]:
train_data, train_labels, _ = prepare_images(train_data, train_folder)
print(train_data.shape, train_labels.shape)
Started preparing images for convnet...
Completed. Images cropped, resized and grayscaled
(33401, 32, 32, 1) (33401, 6)
In [21]:
test_data, test_labels, test_filenames = prepare_images(test_data, test_folder)
print(test_data.shape, test_labels.shape)
Started preparing images for convnet...
Completed. Images cropped, resized and grayscaled
(13068, 32, 32, 1) (13068, 6)
In [22]:
extra_data, extra_labels, _ = prepare_images(extra_data, extra_folder)
print(extra_data.shape, extra_data.shape)
Started preparing images for convnet...
Completed. Images cropped, resized and grayscaled
(50000, 32, 32, 1) (50000, 32, 32, 1)
In [23]:
print(extra_labels[:10])
[[ 3  4  7  8 10 10]
 [ 2  7  1 10 10 10]
 [ 3  1  7  4 10 10]
 [ 2  3  0 10 10 10]
 [ 3  2  8  8 10 10]
 [ 2  3  1 10 10 10]
 [ 3  1  7  0 10 10]
 [ 2  8  1 10 10 10]
 [ 2  5  6 10 10 10]
 [ 3  4  4  4 10 10]]
In [ ]:
from sklearn.utils import shuffle

# Here we add new data to our training set from extra set.
# Then we remove this part from memory to free it
train_data_temp = np.concatenate((train_data, extra_data[:40000, :, :, :]))
extra_data_temp = np.delete(extra_data, np.arange(40000), axis=0)

train_labels_temp = np.concatenate((train_labels, extra_labels[:40000]))
extra_labels_temp = np.delete(extra_labels, np.arange(40000), axis=0)

# And then we shuffle all the data we have
train_data_temp, train_labels_temp = shuffle(train_data_temp, train_labels_temp)
extra_data_temp, extra_labels_temp = shuffle(extra_data_temp, extra_labels_temp)
test_data_temp, test_labels_temp, test_filenames_temp = shuffle(test_data, test_labels, test_filenames)

print("\nTrain shapes:", train_data_temp.shape, train_labels_temp.shape)
print("Extra shapes:", extra_data_temp.shape, extra_labels_temp.shape)
print("Test shapes:", test_data_temp.shape, test_labels_temp.shape)
In [25]:
pickle_file = 'SVHN.pickle'

try:
    f = open(pickle_file, 'wb')
    save = {
        'train_data': train_data_temp,
        'train_labels': train_labels_temp,
        'test_data': test_data_temp,
        'test_labels': test_labels_temp,
        'test_filenames': test_filenames_temp,
        'valid_data': extra_data_temp, 
        'valid_labels': extra_labels_temp 
        }
    pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
    f.close()
except Exception as e:
    print('Unable to save data to', pickle_file, ':', e)
    raise
    
statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)
Compressed pickle size: 399965309
In [26]:
from collections import Counter

train_num_length = Counter(train_labels_temp[:,0])
test_num_length = Counter(test_labels_temp[:,0])
extra_num_length = Counter(extra_labels_temp[:,0])
In [27]:
import matplotlib.pyplot as plt

plt.figure(1)

plt.subplot(221)
plt.bar(train_num_length.keys(), train_num_length.values(), align='center')
plt.xticks(train_num_length.keys())
plt.title('Train')
plt.xlabel('Labels')
plt.ylabel('Occurencies')

plt.subplot(222)
plt.bar(test_num_length.keys(), test_num_length.values(), align='center')
plt.xticks(test_num_length.keys())
plt.title('Test')
plt.xlabel('Labels')
plt.ylabel('Occurencies')

plt.subplot(223)
plt.bar(extra_num_length.keys(), extra_num_length.values(), align='center')
plt.xticks(test_num_length.keys())
plt.title('Extra')
plt.xlabel('Labels')
plt.ylabel('Occurencies')

plt.show()
In [28]:
pickle_file = 'SVHN.pickle'

with open(pickle_file, 'rb') as f:
    save = pickle.load(f)
    train_labels = save['train_labels']
    test_labels = save['test_labels']
    valid_labels = save['valid_labels']
    del save
In [29]:
from collections import Counter

# Remove classes of empty labels
train_digits = Counter(train_labels.flatten()[np.where(train_labels.flatten() != 10)])
test_digits = Counter(test_labels.flatten()[np.where(test_labels.flatten() != 10)])
valid_digits = Counter(valid_labels.flatten()[np.where(valid_labels.flatten() != 10)])
In [30]:
import matplotlib.pyplot as plt
%matplotlib inline

f, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(20, 5))

ax1.bar(train_digits.keys(), train_digits.values(), align='center')
ax1.set_xticks(train_digits.keys())
ax1.set_title('Train')
ax1.set_xlabel('Labels')
ax1.set_ylabel('Occurencies')

ax2.bar(test_digits.keys(), test_digits.values(), align='center')
ax2.set_xticks(test_digits.keys())
ax2.set_title('Test')
ax2.set_xlabel('Labels')
ax2.set_ylabel('Occurencies')

ax3.bar(valid_digits.keys(), valid_digits.values(), align='center')
ax3.set_xticks(valid_digits.keys())
ax3.set_title('Validation')
ax3.set_xlabel('Labels')
ax3.set_ylabel('Occurencies')

plt.show()
In [1]:
from __future__ import print_function
import numpy as np
import tensorflow as tf
from sklearn.cross_validation import train_test_split
from six.moves import cPickle as pickle
from six.moves import range
from keras.models import Model
from keras.layers import Input, Dense, TimeDistributed
from keras.layers import LSTM
from keras.utils import np_utils
/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
Using TensorFlow backend.
In [2]:
pickle_file = 'SVHN.pickle'

with open(pickle_file, 'rb') as f:
    save = pickle.load(f)
    train_dataset = save['train_data']
    train_labels = save['train_labels']
    test_dataset = save['test_data']
    test_labels = save['test_labels']
    test_filenames = save['test_filenames']
    valid_dataset = save['valid_data']
    valid_labels = save['valid_labels']
    del save
    
    print('Training set', train_dataset.shape, train_labels.shape)
    print('Validation set', valid_dataset.shape, valid_labels.shape)
    print('Test set', test_dataset.shape, test_labels.shape)
Training set (73401, 32, 32, 1) (73401, 6)
Validation set (10000, 32, 32, 1) (10000, 6)
Test set (13068, 32, 32, 1) (13068, 6)
In [3]:
dataset_size =73401
def createSequences():
    #dataset = np.ndarray(shape=(dataset_size, image_height, 84),dtype=np.float32)

    data_labels1 = []
    data_labels2 = []
    data_labels3 = []
    data_labels4 = []
    data_labels5 = []
    data_labels6 = []

    i = 0
    w = 0
    while i < (dataset_size):
     
        #temp = np.hstack([X_train1[i]])
        
        
        #dataset[i, :, :] = temp
        
        #temp = np.hstack([XX[i],XY[i],XZ[i]])
        
        #dataset[i, :, :] = temp
        temp_str1 = (train_labels[i][0])
        temp_str2 = (train_labels[i][1])
        temp_str3 = (train_labels[i][2])
        temp_str4 = (train_labels[i][3])
        temp_str5 = (train_labels[i][4])
        temp_str6 = (train_labels[i][5])
       
        
        data_labels1.append(temp_str1)
        data_labels2.append(temp_str2)
        data_labels3.append(temp_str3)
        data_labels4.append(temp_str4)
        data_labels5.append(temp_str5)
        data_labels6.append(temp_str6)
        
        
        w += 1
        i += 1
        np.array(data_labels1)
        np.array(data_labels2)
        np.array(data_labels3)
        np.array(data_labels4)
        np.array(data_labels5)
        np.array(data_labels6)

    return data_labels1,data_labels2,data_labels3,data_labels4,data_labels5,data_labels6

y1,y2,y3,y4,y5,y6 = createSequences()
from keras.utils import np_utils



print(len(y1))
print(y1[1])
73401
4
In [4]:
dataset_size =13068
def createSequences():
    #dataset = np.ndarray(shape=(dataset_size, image_height, 84),dtype=np.float32)

    data_labels_test1 = []
    data_labels_test2 = []
    data_labels_test3 = []
    data_labels_test4 = []
    data_labels_test5 = []
    data_labels_test6 = []

    i = 0
    w = 0
    while i < (dataset_size):
     
        #temp = np.hstack([X_train1[i]])
        
        
        #dataset[i, :, :] = temp
        
        #temp = np.hstack([XX[i],XY[i],XZ[i]])
        
        #dataset[i, :, :] = temp
        temp_str1 = (test_labels[i][0])
        temp_str2 = (test_labels[i][1])
        temp_str3 = (test_labels[i][2])
        temp_str4 = (test_labels[i][3])
        temp_str5 = (test_labels[i][4])
        temp_str6 = (test_labels[i][5])
       
        
        data_labels_test1.append(temp_str1)
        data_labels_test2.append(temp_str2)
        data_labels_test3.append(temp_str3)
        data_labels_test4.append(temp_str4)
        data_labels_test5.append(temp_str5)
        data_labels_test6.append(temp_str6)
        
        
        w += 1
        i += 1
        np.array(data_labels_test1)
        np.array(data_labels_test2)
        np.array(data_labels_test3)
        np.array(data_labels_test4)
        np.array(data_labels_test5)
        np.array(data_labels_test6)

    return data_labels_test1,data_labels_test2,data_labels_test3,data_labels_test4,data_labels_test5,data_labels_test6

y_1,y_2,y_3,y_4,y_5,y_6 = createSequences()
from keras.utils import np_utils



print(len(y_1))
print(y_1[1])
13068
3
In [5]:
Y1 = np_utils.to_categorical(y1, 11)
Y2 = np_utils.to_categorical(y2, 11)
Y3 = np_utils.to_categorical(y3, 11)
Y4 = np_utils.to_categorical(y4, 11)
Y5 = np_utils.to_categorical(y5, 11)
Y6 = np_utils.to_categorical(y6, 11)
In [6]:
Y_1 = np_utils.to_categorical(y_1, 11)
Y_2 = np_utils.to_categorical(y_2, 11)
Y_3 = np_utils.to_categorical(y_3, 11)
Y_4 = np_utils.to_categorical(y_4, 11)
Y_5 = np_utils.to_categorical(y_5, 11)
Y_6 = np_utils.to_categorical(y_6, 11)
In [7]:
X_test1=test_dataset
X_train1=train_dataset
In [8]:
X_train1=train_dataset
#X_train1 /= 255
In [9]:
from keras.layers import Input, Dense
from keras.models import Model
from keras.layers import Dropout
from keras.layers import Flatten
from keras.layers.convolutional import Convolution2D
from keras.layers.convolutional import MaxPooling2D


inputs = Input(shape=(32,32,1))
conv = Convolution2D(32, 5, 5, border_mode='same', input_shape=( 32,32,1), activation='relu')
conv1 = Convolution2D(64, 5, 5, border_mode='same',  activation='relu')
conv2= Convolution2D(128, 5, 5, border_mode='same',  activation='relu')
x=conv(inputs)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv1(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv2(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
#model.add(Convolution2D(64, 5,5, border_mode='same', activation='relu'))

#model.add(MaxPooling2D(pool_size=(2, 2)))

#model.add(Convolution2D(128, 5, 5, border_mode='same', activation='relu'))

#model.add(MaxPooling2D(pool_size=(2, 2)))
x=Flatten()(x)
x = Dense(1064, activation='relu')(x)
x = Dense(800, activation='relu')(x)
x = Dense(600, activation='relu')(x)
x = Dense(400, activation='relu')(x)
x = Dense(200, activation='relu')(x)
x = Dense(164, activation='relu')(x)
x = Dense(32, activation='relu')(x)
predictions1 = Dense(11, activation='softmax')(x)
predictions2 = Dense(11, activation='softmax')(x)
predictions3 = Dense(11, activation='softmax')(x)
predictions4 = Dense(11, activation='softmax')(x)
predictions5 = Dense(11, activation='softmax')(x)
predictions6= Dense(11, activation='softmax')(x)

model = Model(input=inputs, output=[predictions1,predictions2,predictions3,predictions4,predictions5,predictions6])
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
In [8]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
73401/73401 [==============================] - 258s - loss: 5.4639 - dense_8_loss: 0.4031 - dense_9_loss: 1.5543 - dense_10_loss: 2.0018 - dense_11_loss: 1.1964 - dense_12_loss: 0.2711 - dense_13_loss: 0.0372 - dense_8_acc: 0.8570 - dense_9_acc: 0.4529 - dense_10_acc: 0.2592 - dense_11_acc: 0.6087 - dense_12_acc: 0.9395 - dense_13_acc: 0.9978   
Epoch 2/10
73401/73401 [==============================] - 268s - loss: 2.9500 - dense_8_loss: 0.1692 - dense_9_loss: 0.7638 - dense_10_loss: 1.1042 - dense_11_loss: 0.7277 - dense_12_loss: 0.1778 - dense_13_loss: 0.0072 - dense_8_acc: 0.9523 - dense_9_acc: 0.7539 - dense_10_acc: 0.6135 - dense_11_acc: 0.7567 - dense_12_acc: 0.9504 - dense_13_acc: 0.9995   
Epoch 3/10
73401/73401 [==============================] - 269s - loss: 2.1634 - dense_8_loss: 0.1574 - dense_9_loss: 0.5501 - dense_10_loss: 0.7513 - dense_11_loss: 0.5407 - dense_12_loss: 0.1559 - dense_13_loss: 0.0080 - dense_8_acc: 0.9575 - dense_9_acc: 0.8440 - dense_10_acc: 0.7619 - dense_11_acc: 0.8350 - dense_12_acc: 0.9559 - dense_13_acc: 0.9995   
Epoch 4/10
73401/73401 [==============================] - 269s - loss: 1.9512 - dense_8_loss: 0.1518 - dense_9_loss: 0.5087 - dense_10_loss: 0.6556 - dense_11_loss: 0.4831 - dense_12_loss: 0.1438 - dense_13_loss: 0.0081 - dense_8_acc: 0.9584 - dense_9_acc: 0.8614 - dense_10_acc: 0.8029 - dense_11_acc: 0.8570 - dense_12_acc: 0.9596 - dense_13_acc: 0.9995   
Epoch 5/10
73401/73401 [==============================] - 271s - loss: 1.9016 - dense_8_loss: 0.1603 - dense_9_loss: 0.4969 - dense_10_loss: 0.6309 - dense_11_loss: 0.4590 - dense_12_loss: 0.1465 - dense_13_loss: 0.0081 - dense_8_acc: 0.9584 - dense_9_acc: 0.8705 - dense_10_acc: 0.8118 - dense_11_acc: 0.8680 - dense_12_acc: 0.9602 - dense_13_acc: 0.9995   
Epoch 6/10
73401/73401 [==============================] - 271s - loss: 1.8901 - dense_8_loss: 0.1654 - dense_9_loss: 0.4945 - dense_10_loss: 0.6179 - dense_11_loss: 0.4564 - dense_12_loss: 0.1477 - dense_13_loss: 0.0081 - dense_8_acc: 0.9568 - dense_9_acc: 0.8717 - dense_10_acc: 0.8198 - dense_11_acc: 0.8708 - dense_12_acc: 0.9605 - dense_13_acc: 0.9995   
Epoch 7/10
73401/73401 [==============================] - 275s - loss: 1.9133 - dense_8_loss: 0.1719 - dense_9_loss: 0.5073 - dense_10_loss: 0.6221 - dense_11_loss: 0.4604 - dense_12_loss: 0.1434 - dense_13_loss: 0.0081 - dense_8_acc: 0.9552 - dense_9_acc: 0.8712 - dense_10_acc: 0.8234 - dense_11_acc: 0.8699 - dense_12_acc: 0.9613 - dense_13_acc: 0.9995   
Epoch 8/10
73401/73401 [==============================] - 271s - loss: 1.9076 - dense_8_loss: 0.1664 - dense_9_loss: 0.5097 - dense_10_loss: 0.6170 - dense_11_loss: 0.4592 - dense_12_loss: 0.1472 - dense_13_loss: 0.0081 - dense_8_acc: 0.9552 - dense_9_acc: 0.8720 - dense_10_acc: 0.8233 - dense_11_acc: 0.8700 - dense_12_acc: 0.9611 - dense_13_acc: 0.9995   
Epoch 9/10
73401/73401 [==============================] - 271s - loss: 1.9680 - dense_8_loss: 0.1847 - dense_9_loss: 0.5297 - dense_10_loss: 0.6261 - dense_11_loss: 0.4693 - dense_12_loss: 0.1500 - dense_13_loss: 0.0081 - dense_8_acc: 0.9528 - dense_9_acc: 0.8695 - dense_10_acc: 0.8281 - dense_11_acc: 0.8697 - dense_12_acc: 0.9602 - dense_13_acc: 0.9995   
Epoch 10/10
73401/73401 [==============================] - 278s - loss: 1.9792 - dense_8_loss: 0.1836 - dense_9_loss: 0.5435 - dense_10_loss: 0.6265 - dense_11_loss: 0.4653 - dense_12_loss: 0.1521 - dense_13_loss: 0.0081 - dense_8_acc: 0.9527 - dense_9_acc: 0.8688 - dense_10_acc: 0.8264 - dense_11_acc: 0.8696 - dense_12_acc: 0.9601 - dense_13_acc: 0.9995   
Out[8]:
<keras.callbacks.History at 0x7fa9149eca90>
In [10]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
73401/73401 [==============================] - 267s - loss: 7.4978 - dense_8_loss: 1.0555 - dense_9_loss: 2.0114 - dense_10_loss: 2.3644 - dense_11_loss: 1.7013 - dense_12_loss: 0.3574 - dense_13_loss: 0.0078 - dense_8_acc: 0.4923 - dense_9_acc: 0.2775 - dense_10_acc: 0.1333 - dense_11_acc: 0.5380 - dense_12_acc: 0.9406 - dense_13_acc: 0.9995   
Epoch 2/10
73401/73401 [==============================] - 269s - loss: 6.6467 - dense_8_loss: 0.6997 - dense_9_loss: 1.9795 - dense_10_loss: 2.2118 - dense_11_loss: 1.4294 - dense_12_loss: 0.3181 - dense_13_loss: 0.0081 - dense_8_acc: 0.7131 - dense_9_acc: 0.2866 - dense_10_acc: 0.2014 - dense_11_acc: 0.5663 - dense_12_acc: 0.9406 - dense_13_acc: 0.9995   
Epoch 3/10
73401/73401 [==============================] - 269s - loss: 6.0624 - dense_8_loss: 0.5279 - dense_9_loss: 1.7892 - dense_10_loss: 2.1433 - dense_11_loss: 1.2915 - dense_12_loss: 0.3025 - dense_13_loss: 0.0081 - dense_8_acc: 0.7999 - dense_9_acc: 0.3813 - dense_10_acc: 0.2295 - dense_11_acc: 0.5824 - dense_12_acc: 0.9406 - dense_13_acc: 0.9995   
Epoch 4/10
73401/73401 [==============================] - 277s - loss: 5.3435 - dense_8_loss: 0.3673 - dense_9_loss: 1.5607 - dense_10_loss: 1.9500 - dense_11_loss: 1.1996 - dense_12_loss: 0.2578 - dense_13_loss: 0.0081 - dense_8_acc: 0.8673 - dense_9_acc: 0.4689 - dense_10_acc: 0.3021 - dense_11_acc: 0.5968 - dense_12_acc: 0.9413 - dense_13_acc: 0.9995   
Epoch 5/10
73401/73401 [==============================] - 273s - loss: 4.6603 - dense_8_loss: 0.2977 - dense_9_loss: 1.2786 - dense_10_loss: 1.6963 - dense_11_loss: 1.1424 - dense_12_loss: 0.2373 - dense_13_loss: 0.0081 - dense_8_acc: 0.8984 - dense_9_acc: 0.5639 - dense_10_acc: 0.3933 - dense_11_acc: 0.6109 - dense_12_acc: 0.9428 - dense_13_acc: 0.9995   
Epoch 6/10
73401/73401 [==============================] - 271s - loss: 4.1448 - dense_8_loss: 0.2596 - dense_9_loss: 1.1081 - dense_10_loss: 1.4790 - dense_11_loss: 1.0667 - dense_12_loss: 0.2234 - dense_13_loss: 0.0080 - dense_8_acc: 0.9122 - dense_9_acc: 0.6244 - dense_10_acc: 0.4738 - dense_11_acc: 0.6372 - dense_12_acc: 0.9435 - dense_13_acc: 0.9995   
Epoch 7/10
73401/73401 [==============================] - 277s - loss: 3.7891 - dense_8_loss: 0.2390 - dense_9_loss: 1.0148 - dense_10_loss: 1.3355 - dense_11_loss: 0.9744 - dense_12_loss: 0.2173 - dense_13_loss: 0.0081 - dense_8_acc: 0.9210 - dense_9_acc: 0.6635 - dense_10_acc: 0.5287 - dense_11_acc: 0.6628 - dense_12_acc: 0.9444 - dense_13_acc: 0.9995   
Epoch 8/10
73401/73401 [==============================] - 275s - loss: 3.5304 - dense_8_loss: 0.2303 - dense_9_loss: 0.9450 - dense_10_loss: 1.2275 - dense_11_loss: 0.9084 - dense_12_loss: 0.2111 - dense_13_loss: 0.0081 - dense_8_acc: 0.9262 - dense_9_acc: 0.6902 - dense_10_acc: 0.5707 - dense_11_acc: 0.6861 - dense_12_acc: 0.9448 - dense_13_acc: 0.9995   
Epoch 9/10
73401/73401 [==============================] - 273s - loss: 3.3524 - dense_8_loss: 0.2252 - dense_9_loss: 0.8948 - dense_10_loss: 1.1426 - dense_11_loss: 0.8720 - dense_12_loss: 0.2097 - dense_13_loss: 0.0081 - dense_8_acc: 0.9287 - dense_9_acc: 0.7087 - dense_10_acc: 0.6013 - dense_11_acc: 0.6976 - dense_12_acc: 0.9453 - dense_13_acc: 0.9995   
Epoch 10/10
73401/73401 [==============================] - 277s - loss: 3.1855 - dense_8_loss: 0.2141 - dense_9_loss: 0.8472 - dense_10_loss: 1.0805 - dense_11_loss: 0.8317 - dense_12_loss: 0.2039 - dense_13_loss: 0.0081 - dense_8_acc: 0.9321 - dense_9_acc: 0.7256 - dense_10_acc: 0.6276 - dense_11_acc: 0.7108 - dense_12_acc: 0.9457 - dense_13_acc: 0.9995   
Out[10]:
<keras.callbacks.History at 0x7fa9143aadd0>
In [11]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
73401/73401 [==============================] - 272s - loss: 3.0484 - dense_8_loss: 0.2103 - dense_9_loss: 0.8026 - dense_10_loss: 1.0264 - dense_11_loss: 0.8004 - dense_12_loss: 0.2005 - dense_13_loss: 0.0081 - dense_8_acc: 0.9344 - dense_9_acc: 0.7396 - dense_10_acc: 0.6522 - dense_11_acc: 0.7266 - dense_12_acc: 0.9466 - dense_13_acc: 0.9995   
Epoch 2/10
73401/73401 [==============================] - 275s - loss: 2.9335 - dense_8_loss: 0.2041 - dense_9_loss: 0.7699 - dense_10_loss: 0.9811 - dense_11_loss: 0.7708 - dense_12_loss: 0.1995 - dense_13_loss: 0.0081 - dense_8_acc: 0.9366 - dense_9_acc: 0.7543 - dense_10_acc: 0.6693 - dense_11_acc: 0.7390 - dense_12_acc: 0.9471 - dense_13_acc: 0.9995   
Epoch 3/10
73401/73401 [==============================] - 273s - loss: 2.8771 - dense_8_loss: 0.2052 - dense_9_loss: 0.7540 - dense_10_loss: 0.9582 - dense_11_loss: 0.7518 - dense_12_loss: 0.1998 - dense_13_loss: 0.0081 - dense_8_acc: 0.9364 - dense_9_acc: 0.7629 - dense_10_acc: 0.6797 - dense_11_acc: 0.7461 - dense_12_acc: 0.9473 - dense_13_acc: 0.9995   
Epoch 4/10
73401/73401 [==============================] - 273s - loss: 2.7945 - dense_8_loss: 0.2016 - dense_9_loss: 0.7327 - dense_10_loss: 0.9273 - dense_11_loss: 0.7249 - dense_12_loss: 0.1999 - dense_13_loss: 0.0081 - dense_8_acc: 0.9375 - dense_9_acc: 0.7706 - dense_10_acc: 0.6926 - dense_11_acc: 0.7580 - dense_12_acc: 0.9474 - dense_13_acc: 0.9995   
Epoch 5/10
73401/73401 [==============================] - 274s - loss: 2.7332 - dense_8_loss: 0.1996 - dense_9_loss: 0.7242 - dense_10_loss: 0.9011 - dense_11_loss: 0.7024 - dense_12_loss: 0.1978 - dense_13_loss: 0.0081 - dense_8_acc: 0.9389 - dense_9_acc: 0.7743 - dense_10_acc: 0.7026 - dense_11_acc: 0.7676 - dense_12_acc: 0.9477 - dense_13_acc: 0.9995   
Epoch 6/10
73401/73401 [==============================] - 273s - loss: 2.6880 - dense_8_loss: 0.1980 - dense_9_loss: 0.7110 - dense_10_loss: 0.8857 - dense_11_loss: 0.6895 - dense_12_loss: 0.1956 - dense_13_loss: 0.0081 - dense_8_acc: 0.9384 - dense_9_acc: 0.7808 - dense_10_acc: 0.7068 - dense_11_acc: 0.7739 - dense_12_acc: 0.9486 - dense_13_acc: 0.9995   
Epoch 7/10
73401/73401 [==============================] - 275s - loss: 2.6470 - dense_8_loss: 0.1944 - dense_9_loss: 0.7087 - dense_10_loss: 0.8685 - dense_11_loss: 0.6739 - dense_12_loss: 0.1934 - dense_13_loss: 0.0081 - dense_8_acc: 0.9404 - dense_9_acc: 0.7819 - dense_10_acc: 0.7169 - dense_11_acc: 0.7781 - dense_12_acc: 0.9487 - dense_13_acc: 0.9995   
Epoch 8/10
73401/73401 [==============================] - 273s - loss: 2.6197 - dense_8_loss: 0.1949 - dense_9_loss: 0.6939 - dense_10_loss: 0.8642 - dense_11_loss: 0.6640 - dense_12_loss: 0.1947 - dense_13_loss: 0.0081 - dense_8_acc: 0.9396 - dense_9_acc: 0.7876 - dense_10_acc: 0.7184 - dense_11_acc: 0.7838 - dense_12_acc: 0.9489 - dense_13_acc: 0.9995   
Epoch 9/10
73401/73401 [==============================] - 277s - loss: 2.5796 - dense_8_loss: 0.1948 - dense_9_loss: 0.6823 - dense_10_loss: 0.8466 - dense_11_loss: 0.6555 - dense_12_loss: 0.1923 - dense_13_loss: 0.0081 - dense_8_acc: 0.9400 - dense_9_acc: 0.7915 - dense_10_acc: 0.7240 - dense_11_acc: 0.7877 - dense_12_acc: 0.9483 - dense_13_acc: 0.9995   
Epoch 10/10
73401/73401 [==============================] - 276s - loss: 2.5656 - dense_8_loss: 0.1941 - dense_9_loss: 0.6825 - dense_10_loss: 0.8428 - dense_11_loss: 0.6442 - dense_12_loss: 0.1939 - dense_13_loss: 0.0081 - dense_8_acc: 0.9404 - dense_9_acc: 0.7919 - dense_10_acc: 0.7266 - dense_11_acc: 0.7914 - dense_12_acc: 0.9488 - dense_13_acc: 0.9995   
Out[11]:
<keras.callbacks.History at 0x7fa9143aa790>
In [10]:
from keras.layers import Input, Dense
from keras.models import Model
from keras.layers import Dropout
from keras.layers import Flatten
from keras.layers.convolutional import Convolution2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers import Dropout

inputs = Input(shape=(32,32,1))
conv = Convolution2D(32, 5, 5, border_mode='same', input_shape=( 32,32,1), activation='relu')
conv1 = Convolution2D(64, 5, 5, border_mode='same',  activation='relu')
conv2= Convolution2D(128, 5, 5, border_mode='same',  activation='relu')
conv3= Convolution2D(256, 5, 5, border_mode='same',  activation='relu')
x=conv(inputs)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv1(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv2(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv3(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
#model.add(Convolution2D(64, 5,5, border_mode='same', activation='relu'))

#model.add(MaxPooling2D(pool_size=(2, 2)))

#model.add(Convolution2D(128, 5, 5, border_mode='same', activation='relu'))

#model.add(MaxPooling2D(pool_size=(2, 2)))
x=Flatten()(x)
x=Dropout(0.2)(x)
x = Dense(1064, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(800, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(600, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(400, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(200, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(164, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(32, activation='relu')(x)
x=Dropout(0.2)(x)
predictions1 = Dense(11, activation='softmax')(x)
predictions2 = Dense(11, activation='softmax')(x)
predictions3 = Dense(11, activation='softmax')(x)
predictions4 = Dense(11, activation='softmax')(x)
predictions5 = Dense(11, activation='softmax')(x)
predictions6= Dense(11, activation='softmax')(x)

model = Model(input=inputs, output=[predictions1,predictions2,predictions3,predictions4,predictions5,predictions6])
model.compile(optimizer='rmsprop',loss='categorical_crossentropy',metrics=['accuracy'])
In [7]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
73401/73401 [==============================] - 257s - loss: 5.3364 - dense_8_loss: 0.3958 - dense_9_loss: 1.5783 - dense_10_loss: 1.8886 - dense_11_loss: 1.1997 - dense_12_loss: 0.2575 - dense_13_loss: 0.0166 - dense_8_acc: 0.8641 - dense_9_acc: 0.4466 - dense_10_acc: 0.3230 - dense_11_acc: 0.6045 - dense_12_acc: 0.9416 - dense_13_acc: 0.9987   
Epoch 2/10
73401/73401 [==============================] - 262s - loss: 2.8950 - dense_8_loss: 0.1785 - dense_9_loss: 0.7934 - dense_10_loss: 0.9993 - dense_11_loss: 0.7281 - dense_12_loss: 0.1878 - dense_13_loss: 0.0080 - dense_8_acc: 0.9506 - dense_9_acc: 0.7427 - dense_10_acc: 0.6643 - dense_11_acc: 0.7565 - dense_12_acc: 0.9485 - dense_13_acc: 0.9995   
Epoch 3/10
73401/73401 [==============================] - 264s - loss: 2.1183 - dense_8_loss: 0.1582 - dense_9_loss: 0.5576 - dense_10_loss: 0.7179 - dense_11_loss: 0.5207 - dense_12_loss: 0.1559 - dense_13_loss: 0.0081 - dense_8_acc: 0.9567 - dense_9_acc: 0.8368 - dense_10_acc: 0.7756 - dense_11_acc: 0.8407 - dense_12_acc: 0.9565 - dense_13_acc: 0.9995   
Epoch 4/10
73401/73401 [==============================] - 267s - loss: 1.9461 - dense_8_loss: 0.1572 - dense_9_loss: 0.5043 - dense_10_loss: 0.6525 - dense_11_loss: 0.4741 - dense_12_loss: 0.1499 - dense_13_loss: 0.0081 - dense_8_acc: 0.9581 - dense_9_acc: 0.8609 - dense_10_acc: 0.8047 - dense_11_acc: 0.8603 - dense_12_acc: 0.9587 - dense_13_acc: 0.9995   
Epoch 5/10
73401/73401 [==============================] - 266s - loss: 1.9383 - dense_8_loss: 0.1649 - dense_9_loss: 0.5054 - dense_10_loss: 0.6416 - dense_11_loss: 0.4675 - dense_12_loss: 0.1507 - dense_13_loss: 0.0081 - dense_8_acc: 0.9563 - dense_9_acc: 0.8635 - dense_10_acc: 0.8130 - dense_11_acc: 0.8646 - dense_12_acc: 0.9593 - dense_13_acc: 0.9995   
Epoch 6/10
73401/73401 [==============================] - 270s - loss: 1.9613 - dense_8_loss: 0.1793 - dense_9_loss: 0.5025 - dense_10_loss: 0.6461 - dense_11_loss: 0.4727 - dense_12_loss: 0.1526 - dense_13_loss: 0.0081 - dense_8_acc: 0.9528 - dense_9_acc: 0.8673 - dense_10_acc: 0.8134 - dense_11_acc: 0.8649 - dense_12_acc: 0.9569 - dense_13_acc: 0.9995   
Epoch 7/10
73401/73401 [==============================] - 274s - loss: 1.9593 - dense_8_loss: 0.1716 - dense_9_loss: 0.5128 - dense_10_loss: 0.6478 - dense_11_loss: 0.4676 - dense_12_loss: 0.1514 - dense_13_loss: 0.0081 - dense_8_acc: 0.9549 - dense_9_acc: 0.8686 - dense_10_acc: 0.8161 - dense_11_acc: 0.8672 - dense_12_acc: 0.9586 - dense_13_acc: 0.9995   
Epoch 8/10
73401/73401 [==============================] - 276s - loss: 1.9896 - dense_8_loss: 0.1772 - dense_9_loss: 0.5234 - dense_10_loss: 0.6507 - dense_11_loss: 0.4751 - dense_12_loss: 0.1550 - dense_13_loss: 0.0081 - dense_8_acc: 0.9540 - dense_9_acc: 0.8686 - dense_10_acc: 0.8183 - dense_11_acc: 0.8665 - dense_12_acc: 0.9588 - dense_13_acc: 0.9995   
Epoch 9/10
73401/73401 [==============================] - 274s - loss: 1.9614 - dense_8_loss: 0.1773 - dense_9_loss: 0.5169 - dense_10_loss: 0.6335 - dense_11_loss: 0.4693 - dense_12_loss: 0.1563 - dense_13_loss: 0.0081 - dense_8_acc: 0.9542 - dense_9_acc: 0.8705 - dense_10_acc: 0.8226 - dense_11_acc: 0.8690 - dense_12_acc: 0.9584 - dense_13_acc: 0.9995   
Epoch 10/10
73401/73401 [==============================] - 271s - loss: 2.0177 - dense_8_loss: 0.1836 - dense_9_loss: 0.5507 - dense_10_loss: 0.6493 - dense_11_loss: 0.4709 - dense_12_loss: 0.1551 - dense_13_loss: 0.0081 - dense_8_acc: 0.9527 - dense_9_acc: 0.8701 - dense_10_acc: 0.8218 - dense_11_acc: 0.8693 - dense_12_acc: 0.9586 - dense_13_acc: 0.9995   
Out[7]:
<keras.callbacks.History at 0x7f7a35cf8c90>
In [8]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
73401/73401 [==============================] - 267s - loss: 1.9653 - dense_8_loss: 0.1785 - dense_9_loss: 0.5212 - dense_10_loss: 0.6374 - dense_11_loss: 0.4665 - dense_12_loss: 0.1536 - dense_13_loss: 0.0081 - dense_8_acc: 0.9530 - dense_9_acc: 0.8696 - dense_10_acc: 0.8244 - dense_11_acc: 0.8698 - dense_12_acc: 0.9588 - dense_13_acc: 0.9995   
Epoch 2/10
73401/73401 [==============================] - 269s - loss: 1.9763 - dense_8_loss: 0.1812 - dense_9_loss: 0.5247 - dense_10_loss: 0.6342 - dense_11_loss: 0.4750 - dense_12_loss: 0.1531 - dense_13_loss: 0.0081 - dense_8_acc: 0.9534 - dense_9_acc: 0.8695 - dense_10_acc: 0.8264 - dense_11_acc: 0.8685 - dense_12_acc: 0.9588 - dense_13_acc: 0.9995   
Epoch 3/10
73401/73401 [==============================] - 270s - loss: 2.0268 - dense_8_loss: 0.1825 - dense_9_loss: 0.5383 - dense_10_loss: 0.6725 - dense_11_loss: 0.4713 - dense_12_loss: 0.1541 - dense_13_loss: 0.0081 - dense_8_acc: 0.9521 - dense_9_acc: 0.8690 - dense_10_acc: 0.8233 - dense_11_acc: 0.8695 - dense_12_acc: 0.9592 - dense_13_acc: 0.9995   
Epoch 4/10
73401/73401 [==============================] - 269s - loss: 2.0306 - dense_8_loss: 0.1897 - dense_9_loss: 0.5549 - dense_10_loss: 0.6454 - dense_11_loss: 0.4769 - dense_12_loss: 0.1555 - dense_13_loss: 0.0081 - dense_8_acc: 0.9517 - dense_9_acc: 0.8639 - dense_10_acc: 0.8228 - dense_11_acc: 0.8695 - dense_12_acc: 0.9595 - dense_13_acc: 0.9995   
Epoch 5/10
73401/73401 [==============================] - 273s - loss: 2.0229 - dense_8_loss: 0.1911 - dense_9_loss: 0.5581 - dense_10_loss: 0.6464 - dense_11_loss: 0.4631 - dense_12_loss: 0.1561 - dense_13_loss: 0.0081 - dense_8_acc: 0.9508 - dense_9_acc: 0.8652 - dense_10_acc: 0.8234 - dense_11_acc: 0.8721 - dense_12_acc: 0.9594 - dense_13_acc: 0.9995   
Epoch 6/10
73401/73401 [==============================] - 275s - loss: 2.0394 - dense_8_loss: 0.1927 - dense_9_loss: 0.5521 - dense_10_loss: 0.6578 - dense_11_loss: 0.4733 - dense_12_loss: 0.1554 - dense_13_loss: 0.0081 - dense_8_acc: 0.9535 - dense_9_acc: 0.8660 - dense_10_acc: 0.8235 - dense_11_acc: 0.8725 - dense_12_acc: 0.9599 - dense_13_acc: 0.9995   
Epoch 7/10
73401/73401 [==============================] - 278s - loss: 2.0883 - dense_8_loss: 0.2015 - dense_9_loss: 0.5650 - dense_10_loss: 0.6704 - dense_11_loss: 0.4832 - dense_12_loss: 0.1600 - dense_13_loss: 0.0081 - dense_8_acc: 0.9502 - dense_9_acc: 0.8639 - dense_10_acc: 0.8193 - dense_11_acc: 0.8696 - dense_12_acc: 0.9589 - dense_13_acc: 0.9995   
Epoch 8/10
73401/73401 [==============================] - 276s - loss: 2.1998 - dense_8_loss: 0.2151 - dense_9_loss: 0.5967 - dense_10_loss: 0.7195 - dense_11_loss: 0.5011 - dense_12_loss: 0.1593 - dense_13_loss: 0.0081 - dense_8_acc: 0.9473 - dense_9_acc: 0.8590 - dense_10_acc: 0.8107 - dense_11_acc: 0.8659 - dense_12_acc: 0.9593 - dense_13_acc: 0.9995   
Epoch 9/10
73401/73401 [==============================] - 274s - loss: 2.1101 - dense_8_loss: 0.1971 - dense_9_loss: 0.5800 - dense_10_loss: 0.6799 - dense_11_loss: 0.4875 - dense_12_loss: 0.1575 - dense_13_loss: 0.0081 - dense_8_acc: 0.9507 - dense_9_acc: 0.8589 - dense_10_acc: 0.8145 - dense_11_acc: 0.8661 - dense_12_acc: 0.9592 - dense_13_acc: 0.9995   
Epoch 10/10
73401/73401 [==============================] - 273s - loss: 2.2918 - dense_8_loss: 0.2280 - dense_9_loss: 0.6290 - dense_10_loss: 0.7493 - dense_11_loss: 0.5124 - dense_12_loss: 0.1650 - dense_13_loss: 0.0081 - dense_8_acc: 0.9465 - dense_9_acc: 0.8520 - dense_10_acc: 0.8065 - dense_11_acc: 0.8629 - dense_12_acc: 0.9583 - dense_13_acc: 0.9995   
Out[8]:
<keras.callbacks.History at 0x7f7a356c7d90>
In [14]:
print('started............')
scores = model.evaluate(X_test1, [Y_1,Y_2,Y_3,Y_4,Y_5,Y_6], verbose=0)
print('Test loss and Test Accuracy', scores)
started............
Test loss and Test Accuracy [2.1286202774596643, 0.32151728484894765, 0.71212437768065673, 0.80010225081757591, 0.25572110039649809, 0.036688342616980688, 0.002466922620768061, 0.94773492500765233, 0.84963269052353696, 0.81848790941524474, 0.93954698501977496, 0.99142944597490057, 0.99984695439240889]
In [40]:
print('started............')
scores = model.evaluate(X_test1, [Y_1,Y_2,Y_3,Y_4,Y_5,Y_6], verbose=0)
print('Test loss and Test Accuracy', scores)
started............
Test loss and Test Accuracy [2.1286202774596643, 0.32151728484894765, 0.71212437768065673, 0.80010225081757591, 0.25572110039649809, 0.036688342616980688, 0.002466922620768061, 0.94773492500765233, 0.84963269052353696, 0.81848790941524474, 0.93954698501977496, 0.99142944597490057, 0.99984695439240889]
In [10]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
73401/73401 [==============================] - 260s - loss: 5.4393 - dense_8_loss: 0.3977 - dense_9_loss: 1.5008 - dense_10_loss: 1.9881 - dense_11_loss: 1.2481 - dense_12_loss: 0.2618 - dense_13_loss: 0.0428 - dense_8_acc: 0.8649 - dense_9_acc: 0.4823 - dense_10_acc: 0.2753 - dense_11_acc: 0.5863 - dense_12_acc: 0.9404 - dense_13_acc: 0.9973   
Epoch 2/10
73401/73401 [==============================] - 264s - loss: 3.0107 - dense_8_loss: 0.1771 - dense_9_loss: 0.6764 - dense_10_loss: 1.1140 - dense_11_loss: 0.8489 - dense_12_loss: 0.1868 - dense_13_loss: 0.0075 - dense_8_acc: 0.9517 - dense_9_acc: 0.7928 - dense_10_acc: 0.6136 - dense_11_acc: 0.7043 - dense_12_acc: 0.9476 - dense_13_acc: 0.9995   
Epoch 3/10
73401/73401 [==============================] - 265s - loss: 2.2048 - dense_8_loss: 0.1584 - dense_9_loss: 0.5243 - dense_10_loss: 0.7660 - dense_11_loss: 0.5802 - dense_12_loss: 0.1680 - dense_13_loss: 0.0080 - dense_8_acc: 0.9575 - dense_9_acc: 0.8514 - dense_10_acc: 0.7574 - dense_11_acc: 0.8160 - dense_12_acc: 0.9535 - dense_13_acc: 0.9995   
Epoch 4/10
73401/73401 [==============================] - 270s - loss: 1.9486 - dense_8_loss: 0.1504 - dense_9_loss: 0.4807 - dense_10_loss: 0.6640 - dense_11_loss: 0.4919 - dense_12_loss: 0.1536 - dense_13_loss: 0.0081 - dense_8_acc: 0.9592 - dense_9_acc: 0.8688 - dense_10_acc: 0.7971 - dense_11_acc: 0.8531 - dense_12_acc: 0.9561 - dense_13_acc: 0.9995   
Epoch 5/10
73401/73401 [==============================] - 271s - loss: 1.9307 - dense_8_loss: 0.1656 - dense_9_loss: 0.4808 - dense_10_loss: 0.6482 - dense_11_loss: 0.4750 - dense_12_loss: 0.1530 - dense_13_loss: 0.0081 - dense_8_acc: 0.9577 - dense_9_acc: 0.8734 - dense_10_acc: 0.8096 - dense_11_acc: 0.8608 - dense_12_acc: 0.9561 - dense_13_acc: 0.9995   
Epoch 6/10
73401/73401 [==============================] - 273s - loss: 1.8898 - dense_8_loss: 0.1684 - dense_9_loss: 0.4729 - dense_10_loss: 0.6252 - dense_11_loss: 0.4611 - dense_12_loss: 0.1541 - dense_13_loss: 0.0081 - dense_8_acc: 0.9574 - dense_9_acc: 0.8774 - dense_10_acc: 0.8194 - dense_11_acc: 0.8683 - dense_12_acc: 0.9576 - dense_13_acc: 0.9995   
Epoch 7/10
73401/73401 [==============================] - 274s - loss: 1.8970 - dense_8_loss: 0.1636 - dense_9_loss: 0.4873 - dense_10_loss: 0.6237 - dense_11_loss: 0.4647 - dense_12_loss: 0.1496 - dense_13_loss: 0.0081 - dense_8_acc: 0.9570 - dense_9_acc: 0.8764 - dense_10_acc: 0.8221 - dense_11_acc: 0.8679 - dense_12_acc: 0.9582 - dense_13_acc: 0.9995   
Epoch 8/10
73401/73401 [==============================] - 274s - loss: 1.8818 - dense_8_loss: 0.1695 - dense_9_loss: 0.4754 - dense_10_loss: 0.6234 - dense_11_loss: 0.4526 - dense_12_loss: 0.1528 - dense_13_loss: 0.0081 - dense_8_acc: 0.9543 - dense_9_acc: 0.8791 - dense_10_acc: 0.8243 - dense_11_acc: 0.8693 - dense_12_acc: 0.9579 - dense_13_acc: 0.9995   
Epoch 9/10
73401/73401 [==============================] - 275s - loss: 1.8872 - dense_8_loss: 0.1679 - dense_9_loss: 0.4923 - dense_10_loss: 0.6207 - dense_11_loss: 0.4492 - dense_12_loss: 0.1490 - dense_13_loss: 0.0081 - dense_8_acc: 0.9563 - dense_9_acc: 0.8787 - dense_10_acc: 0.8266 - dense_11_acc: 0.8744 - dense_12_acc: 0.9586 - dense_13_acc: 0.9995   
Epoch 10/10
73401/73401 [==============================] - 276s - loss: 1.9243 - dense_8_loss: 0.1782 - dense_9_loss: 0.5031 - dense_10_loss: 0.6205 - dense_11_loss: 0.4582 - dense_12_loss: 0.1562 - dense_13_loss: 0.0081 - dense_8_acc: 0.9530 - dense_9_acc: 0.8771 - dense_10_acc: 0.8273 - dense_11_acc: 0.8732 - dense_12_acc: 0.9583 - dense_13_acc: 0.9995   
Out[10]:
<keras.callbacks.History at 0x7fa58ba6ba90>
In [19]:
model.fit(X_train1, [Y1,Y2,Y3,Y4,Y5,Y6])
Epoch 1/10
73401/73401 [==============================] - 257s - loss: 5.3461 - dense_8_loss: 0.3838 - dense_9_loss: 1.5964 - dense_10_loss: 1.8784 - dense_11_loss: 1.2164 - dense_12_loss: 0.2437 - dense_13_loss: 0.0274 - dense_8_acc: 0.8703 - dense_9_acc: 0.4450 - dense_10_acc: 0.3222 - dense_11_acc: 0.5953 - dense_12_acc: 0.9419 - dense_13_acc: 0.9991   
Epoch 2/10
73401/73401 [==============================] - 266s - loss: 2.8154 - dense_8_loss: 0.1707 - dense_9_loss: 0.7719 - dense_10_loss: 0.9508 - dense_11_loss: 0.7208 - dense_12_loss: 0.1939 - dense_13_loss: 0.0074 - dense_8_acc: 0.9540 - dense_9_acc: 0.7509 - dense_10_acc: 0.6794 - dense_11_acc: 0.7610 - dense_12_acc: 0.9462 - dense_13_acc: 0.9995   
Epoch 3/10
73401/73401 [==============================] - 276s - loss: 2.1278 - dense_8_loss: 0.1592 - dense_9_loss: 0.5584 - dense_10_loss: 0.7119 - dense_11_loss: 0.5215 - dense_12_loss: 0.1689 - dense_13_loss: 0.0078 - dense_8_acc: 0.9570 - dense_9_acc: 0.8396 - dense_10_acc: 0.7785 - dense_11_acc: 0.8403 - dense_12_acc: 0.9530 - dense_13_acc: 0.9995   
Epoch 4/10
73401/73401 [==============================] - 275s - loss: 1.9451 - dense_8_loss: 0.1535 - dense_9_loss: 0.5099 - dense_10_loss: 0.6470 - dense_11_loss: 0.4746 - dense_12_loss: 0.1520 - dense_13_loss: 0.0080 - dense_8_acc: 0.9590 - dense_9_acc: 0.8598 - dense_10_acc: 0.8050 - dense_11_acc: 0.8581 - dense_12_acc: 0.9571 - dense_13_acc: 0.9995   
Epoch 5/10
73401/73401 [==============================] - 277s - loss: 1.9296 - dense_8_loss: 0.1591 - dense_9_loss: 0.5099 - dense_10_loss: 0.6335 - dense_11_loss: 0.4675 - dense_12_loss: 0.1515 - dense_13_loss: 0.0081 - dense_8_acc: 0.9588 - dense_9_acc: 0.8644 - dense_10_acc: 0.8143 - dense_11_acc: 0.8631 - dense_12_acc: 0.9588 - dense_13_acc: 0.9995   
Epoch 6/10
73401/73401 [==============================] - 279s - loss: 1.9485 - dense_8_loss: 0.1652 - dense_9_loss: 0.5148 - dense_10_loss: 0.6499 - dense_11_loss: 0.4622 - dense_12_loss: 0.1483 - dense_13_loss: 0.0081 - dense_8_acc: 0.9554 - dense_9_acc: 0.8648 - dense_10_acc: 0.8132 - dense_11_acc: 0.8671 - dense_12_acc: 0.9593 - dense_13_acc: 0.9995   
Epoch 7/10
73401/73401 [==============================] - 283s - loss: 2.0288 - dense_8_loss: 0.1831 - dense_9_loss: 0.5395 - dense_10_loss: 0.6702 - dense_11_loss: 0.4753 - dense_12_loss: 0.1526 - dense_13_loss: 0.0081 - dense_8_acc: 0.9534 - dense_9_acc: 0.8627 - dense_10_acc: 0.8143 - dense_11_acc: 0.8654 - dense_12_acc: 0.9588 - dense_13_acc: 0.9995   
Epoch 8/10
73401/73401 [==============================] - 283s - loss: 2.0234 - dense_8_loss: 0.1849 - dense_9_loss: 0.5518 - dense_10_loss: 0.6499 - dense_11_loss: 0.4756 - dense_12_loss: 0.1531 - dense_13_loss: 0.0081 - dense_8_acc: 0.9528 - dense_9_acc: 0.8619 - dense_10_acc: 0.8172 - dense_11_acc: 0.8640 - dense_12_acc: 0.9583 - dense_13_acc: 0.9995   
Epoch 9/10
73401/73401 [==============================] - 283s - loss: 2.0504 - dense_8_loss: 0.1865 - dense_9_loss: 0.5521 - dense_10_loss: 0.6642 - dense_11_loss: 0.4824 - dense_12_loss: 0.1571 - dense_13_loss: 0.0081 - dense_8_acc: 0.9509 - dense_9_acc: 0.8624 - dense_10_acc: 0.8153 - dense_11_acc: 0.8646 - dense_12_acc: 0.9586 - dense_13_acc: 0.9995   
Epoch 10/10
73401/73401 [==============================] - 284s - loss: 2.0907 - dense_8_loss: 0.1954 - dense_9_loss: 0.5674 - dense_10_loss: 0.6827 - dense_11_loss: 0.4820 - dense_12_loss: 0.1550 - dense_13_loss: 0.0081 - dense_8_acc: 0.9501 - dense_9_acc: 0.8584 - dense_10_acc: 0.8155 - dense_11_acc: 0.8654 - dense_12_acc: 0.9581 - dense_13_acc: 0.9995   
Out[19]:
<keras.callbacks.History at 0x7f6662953e50>
In [20]:
print('started............')
scores = model.evaluate(X_test1, [Y_1,Y_2,Y_3,Y_4,Y_5,Y_6], verbose=0)
print('Test loss and Test Accuracy', scores)
started............
Test loss and Test Accuracy [1.9979013390303169, 0.20649362521951337, 0.67750929346645061, 0.80451120485907712, 0.26272637853846481, 0.044193909798376667, 0.0024669268125372037, 0.94804101624107884, 0.82774716863801512, 0.79025099477820493, 0.93036424856431121, 0.99051117232935415, 0.99984695439240889]

Question 4

Describe how you set up the training and testing data for your model. How does the model perform on a realistic dataset?

The taring data and testing data was of size 32*32 and they consists of one digit to five digits. The training and

testing labels consists of six digits in array where the first digit gives number of digits present in image and the

remaining five digits gives digits in image. Blank was given by '10'

The model on X_train1 was trained for 20 epochs with batch size of 32

The model performaed well on reliastic data with nearly 89% to 100% accuracy

Accuracy of realistic data test set (X_test1) for six classifiers was 0.94804101624107884, 0.82774716863801512, 0.79025099477820493, 0.93036424856431121, 0.99051117232935415, 0.99984695439240889 respectively.

Question 5

What changes did you have to make, if any, to achieve "good" results? Were there any options you explored that made the results worse?

I added third convolution layer 'Conv3' of (256,5,5) along with Max-Pooling layer of 2*2 to acheive good results

Question 6

What were your initial and final results with testing on a realistic dataset? Do you believe your model is doing a good enough job at classifying numbers correctly?

The Final results are better compare to Initial results, initial results for six classifiers are dense_8_acc: 0.9404 - dense_9_acc: 0.7919 - dense_10_acc: 0.7266 - dense_11_acc: 0.7914 - dense_12_acc: 0.9488 - dense_13_acc: 0.9995

The Final results for six classifiers for six digits are dense_8_acc: 0.9501 - dense_9_acc: 0.8584 - dense_10_acc: 0.8155 - dense_11_acc: 0.8654 - dense_12_acc: 0.9581 - dense_13_acc: 0.9995

Clear improvement of accracies of first five classifiers can be seen.

The model is doing well to classify numbers correctly


Step 3: Test a Model on Newly-Captured Images

Take several pictures of numbers that you find around you (at least five), and run them through your classifier on your computer to produce example results. Alternatively (optionally), you can try using OpenCV / SimpleCV / Pygame to capture live images from a webcam and run those through your classifier.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [28]:
from __future__ import print_function
from PIL import Image
from keras.datasets import mnist
from keras.models import Model
from keras.layers import Input, Dense, TimeDistributed
from keras.layers import LSTM
from keras.utils import np_utils
import numpy as np
import pandas as pd
from time import time
from IPython.display import display 
from __future__ import print_function
import matplotlib.pyplot as plt
import numpy as np
import os
import sys
import tarfile
%matplotlib inline
import random
import math
from IPython.display import Image
from scipy import ndimage
from six.moves.urllib.request import urlretrieve
from six.moves import cPickle as pickle
In [33]:
from os import listdir
from os.path import isfile, join
import numpy
import cv2

mypath='resized/'
onlyfiles = [ f for f in listdir(mypath) if isfile(join(mypath,f)) ]

print(onlyfiles)



images = numpy.empty(len(onlyfiles), dtype=object)
for n in range(0, len(onlyfiles)):
  images[n] = cv2.imread( join(mypath,onlyfiles[n]) )

  
['3.jpg', '7.jpg', '0.jpg', '2.jpg', '1.jpg']
In [34]:
import cv2

for i in range (0,5):
  images[i] = cv2.cvtColor(images[i], cv2.COLOR_BGR2GRAY)

Question 7

Choose five candidate images of numbers you took from around you and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult?

Answer- There are certain qualities of image that might make classification difficult

  1. The digits should not overlap
  2. The digits should be almost of same size (shouln't have very big withvery small size of digits)
  3. The digits should be on a good background
  4. The digits should be on a plain background (with no designs, shapes behind)
  5. The background should not be shiny
In [82]:
for i in range (0,5):
 def createSequences():
        dataset = np.ndarray(shape=(1, 32, 32),dtype=np.float32)
        temp = np.hstack([images[i]])
        dataset[0, :, :] = temp
        return dataset


 dataset = createSequences()


 fig=plt.figure()
 plt.imshow(dataset[0])
 plt.show()
 hand_dataset = dataset[0].reshape(1,32,32,1).astype('float32')
 
 
 
 classes = model.predict(hand_dataset, batch_size=2,verbose=0)
 print (classes)
[array([ 0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.])]
[array([ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.])]
[array([ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.])]
[array([ 0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.])]
[array([ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.])]

Question 8

Is your model able to perform equally well on captured pictures or a live camera stream when compared to testing on the realistic dataset?

The model is performing equally well on captured images when compared to realistic data. On SVHN data the accuracy was 0.94804101624107884, 0.82774716863801512, 0.79025099477820493, 0.93036424856431121, 0.99051117232935415, 0.99984695439240889. I tested the trained model on five images captured on camera, the model could identify the handwritten numbers and perfomed well

Optional: Question 9

If necessary, provide documentation for how an interface was built for your model to load and classify newly-acquired images.

Answer: Leave blank if you did not complete this part.


Step 4: Explore an Improvement for a Model

There are many things you can do once you have the basic classifier in place. One example would be to also localize where the numbers are on the image. The SVHN dataset provides bounding boxes that you can tune to train a localizer. Train a regression loss to the coordinates of the bounding box, and then test it.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [4]:
from __future__ import print_function
import matplotlib.pyplot as plt
import numpy as np
import os
import sys
import tarfile
import tensorflow as tf
from IPython.display import Image
from scipy import ndimage
from six.moves.urllib.request import urlretrieve
from six.moves import cPickle as pickle
%matplotlib inline
In [2]:
url = 'http://ufldl.stanford.edu/housenumbers/'
last_percent_reported = None


def download_progress_hook(count, blockSize, totalSize):
    """
    A hook to report the progress of a download. This is mostly intended for users with
    slow internet connections. Reports every 1% change in download progress.
    """
    global last_percent_reported
    percent = int(count * blockSize * 100 / totalSize)

    if last_percent_reported != percent:
        if percent % 5 == 0:
            sys.stdout.write("%s%%" % percent)
            sys.stdout.flush()
        else:
            sys.stdout.write(".")
            sys.stdout.flush()

        last_percent_reported = percent
        
        
def maybe_download(filename, force=False):
    """
    Download a file if not present, and make sure it's the right size.
    """
    if force or not os.path.exists(filename):
        print('Attempting to download:', filename) 
        filename, _ = urlretrieve(url + filename, filename, reporthook=download_progress_hook)
        print('\nDownload Complete!')
    else:
        print(filename, 'is already downloaded. Skipped.')
    return filename
In [3]:
train_filename = maybe_download('train.tar.gz')
train.tar.gz is already downloaded. Skipped.
In [4]:
test_filename = maybe_download('test.tar.gz')
test.tar.gz is already downloaded. Skipped.
In [5]:
extra_filename = maybe_download('extra.tar.gz')
extra.tar.gz is already downloaded. Skipped.
In [6]:
np.random.seed(133)


def maybe_extract(file_, force=False):
    filename = os.path.splitext(os.path.splitext(file_)[0])[0]  # remove .tar.gz
    
    if os.path.isdir(filename) and not force:
        # You may override by setting force=True.
        print('%s is already presented - Skipping extraction of %s.' % (filename, file_))
    else:
        print('Extracting %s file data. Please wait...' % file_)
        tar = tarfile.open(file_)
        sys.stdout.flush()
        tar.extractall()
        tar.close()
        print('File %s is successfully extracted into %s directory.' % (file_, filename))        
    
    return filename
In [7]:
train_folder = maybe_extract(train_filename)
train is already presented - Skipping extraction of train.tar.gz.
In [8]:
test_folder = maybe_extract(test_filename)
test is already presented - Skipping extraction of test.tar.gz.
In [9]:
extra_folder = maybe_extract(extra_filename)
extra is already presented - Skipping extraction of extra.tar.gz.
In [10]:
def remove_anomaly_samples(data, max_class_length = 5):
    """
    Here we remove all data which has class length higher than specified value.
    """
    print("\nDataset size before update:", len(data))
    
    for i in range(len(data)):
        if i < len(data) and len(data[i]['label']) > max_class_length:
            print("\nAnomaly at index %d detected. Class size: %d" % (i, len(data[i]['label'])))
            del data[i]
            
    print("\nDataset after before update:", len(data))            
    return data
In [11]:
import h5py


class DigitStructsWrapper:
    def __init__(self, file_, start_ = 0, end_ = 0):
        self.file_ = h5py.File(file_, 'r')
        self.names = self.file_['digitStruct']['name'][start_:end_] if end_ > 0 else self.file_['digitStruct']['name']
        self.bboxes = self.file_['digitStruct']['bbox'][start_:end_] if end_ > 0 else self.file_['digitStruct']['bbox']
        self.collectionSize = len(self.names)
        print("\n%s file structure contain %d entries" % (file_, self.collectionSize))
        
        
    def bboxHelper(self, keys_):
        """
        Method handles the coding difference when there is exactly one bbox or an array of bbox. 
        """
        if (len(keys_) > 1):
            val = [self.file_[keys_.value[j].item()].value[0][0] for j in range(len(keys_))]
        else:
            val = [keys_.value[0][0]]
        return val

    
    # getBbox returns a dict of data for the n(th) bbox. 
    def getBbox(self, n):
        bbox = {}
        bb = self.bboxes[n].item()
        bbox['height'] = self.bboxHelper(self.file_[bb]["height"])
        bbox['left'] = self.bboxHelper(self.file_[bb]["left"])
        bbox['top'] = self.bboxHelper(self.file_[bb]["top"])
        bbox['width'] = self.bboxHelper(self.file_[bb]["width"])
        bbox['label'] = self.bboxHelper(self.file_[bb]["label"])
        return bbox

    
    def getName(self, n):
        """
        Method returns the filename for the n(th) digitStruct. Since each letter is stored in a structure 
        as array of ANSII char numbers we should convert it back by calling chr function.
        """
        return ''.join([chr(c[0]) for c in self.file_[self.names[n][0]].value])

    
    def getNumberStructure(self,n):
        s = self.getBbox(n)
        s['name']=self.getName(n)
        return s

    def getAllNumbersStructure(self):
        """
        Method returns an array, which contains information about every image.
        This info contains: positions, labels 
        """
        return [self.getNumberStructure(i) for i in range(self.collectionSize)]

    
    
    def getAllNumbersRestructured(self): 
        train_box=list()
        numbersData = self.getAllNumbersStructure()
        print("\nObject structure before transforming: ", numbersData[0])
        remove_anomaly_samples(numbersData)
        
        result = []
        for numData in numbersData:
            metadatas = []
            for i in range(len(numData['height'])):
                metadata = {}
                metadata['height'] = numData['height'][i]
                metadata['label']  = numData['label'][i]
                metadata['left']   = numData['left'][i]
                metadata['top']    = numData['top'][i]
                metadata['width']  = numData['width'][i]
                metadatas.append(metadata)
                
            result.append({ 'boxes':metadatas, 'name':numData["name"] })
            train_box.append(metadatas)
            
        print("\nObject structure after transforming: ", result[0])
        
        return result,train_box
In [12]:
file_ = os.path.join(train_folder, 'digitStruct.mat')

dsf = DigitStructsWrapper(file_)
train_data,train_box = dsf.getAllNumbersRestructured()
train/digitStruct.mat file structure contain 33402 entries

Object structure before transforming:  {'name': '1.png', 'top': [77.0, 81.0], 'label': [1.0, 9.0], 'width': [81.0, 96.0], 'height': [219.0, 219.0], 'left': [246.0, 323.0]}

Dataset size before update: 33402

Anomaly at index 29929 detected. Class size: 6

Dataset after before update: 33401

Object structure after transforming:  {'boxes': [{'width': 81.0, 'top': 77.0, 'label': 1.0, 'left': 246.0, 'height': 219.0}, {'width': 96.0, 'top': 81.0, 'label': 9.0, 'left': 323.0, 'height': 219.0}], 'name': '1.png'}
In [16]:
print(len(train_box))
print((train_box[0]))
print(train_data[0])
33401
[{'width': 81.0, 'top': 77.0, 'label': 1.0, 'left': 246.0, 'height': 219.0}, {'width': 96.0, 'top': 81.0, 'label': 9.0, 'left': 323.0, 'height': 219.0}]
{'boxes': [{'width': 81.0, 'top': 77.0, 'label': 1.0, 'left': 246.0, 'height': 219.0}, {'width': 96.0, 'top': 81.0, 'label': 9.0, 'left': 323.0, 'height': 219.0}], 'name': '1.png'}
In [17]:
file_ = os.path.join(test_folder, 'digitStruct.mat')
dsf = DigitStructsWrapper(file_)
test_data,test_box = dsf.getAllNumbersRestructured()
test/digitStruct.mat file structure contain 13068 entries

Object structure before transforming:  {'name': '1.png', 'top': [7.0], 'label': [5.0], 'width': [19.0], 'height': [30.0], 'left': [43.0]}

Dataset size before update: 13068

Dataset after before update: 13068

Object structure after transforming:  {'boxes': [{'width': 19.0, 'top': 7.0, 'label': 5.0, 'left': 43.0, 'height': 30.0}], 'name': '1.png'}
In [18]:
print(len(test_box))
13068
In [19]:
file_ = os.path.join(extra_folder, 'digitStruct.mat')
dsf = DigitStructsWrapper(file_, 0, 50000)
extra_data,extra_box = dsf.getAllNumbersRestructured()
extra/digitStruct.mat file structure contain 50000 entries

Object structure before transforming:  {'name': '1.png', 'top': [70.0, 41.0, 23.0], 'label': [4.0, 7.0, 8.0], 'width': [38.0, 36.0, 47.0], 'height': [56.0, 56.0, 56.0], 'left': [24.0, 55.0, 79.0]}

Dataset size before update: 50000

Dataset after before update: 50000

Object structure after transforming:  {'boxes': [{'width': 38.0, 'top': 70.0, 'label': 4.0, 'left': 24.0, 'height': 56.0}, {'width': 36.0, 'top': 41.0, 'label': 7.0, 'left': 55.0, 'height': 56.0}, {'width': 47.0, 'top': 23.0, 'label': 8.0, 'left': 79.0, 'height': 56.0}], 'name': '1.png'}
In [20]:
from PIL import Image

def print_data_stats(data, folder):
    data_imgSize = np.ndarray([len(data),2])

    for i in np.arange(len(data)):
        filename = data[i]['name']
        filepath = os.path.join(folder, filename)
        data_imgSize[i, :] = Image.open(filepath).size[:]

    max_w, max_h = np.amax(data_imgSize[:,0]), np.amax(data_imgSize[:,1])
    min_w, min_h = np.amin(data_imgSize[:,0]), np.amin(data_imgSize[:,1])
    mean_w, mean_h = np.mean(data_imgSize[:,0]), np.mean(data_imgSize[:,1])
    print(folder, "max width and height:", max_w, max_h) 
    print(folder, "min width and height:", min_w, min_h)
    print(folder, "mean width and height:", mean_w, mean_h, "\n")
    
    max_w_i, max_h_i = np.where(data_imgSize[:,0] == max_w), np.where(data_imgSize[:,1] == max_h)
    print(folder, "max width indicies:", max_w_i) 
    print(folder, "max height indicies:", max_h_i, "\n")
    
    
    min_w_i, min_h_i = np.where(data_imgSize[:,0] == min_w), np.where(data_imgSize[:,1] == min_h)
    print(folder, "min width indicies:", min_w_i) 
    print(folder, "min height indicies:", min_h_i, "\n***\n")
In [21]:
print_data_stats(train_data, train_folder)
print_data_stats(test_data, test_folder)
print_data_stats(extra_data, extra_folder)
train max width and height: 876.0 501.0
train min width and height: 25.0 12.0
train mean width and height: 128.286338732 57.2139456902 

train max width indicies: (array([  410,  4163, 15855, 30483]),)
train max height indicies: (array([15855]),) 

train min width indicies: (array([9747]),)
train min height indicies: (array([ 1813,  2291,  4829,  5691,  9488,  9747,  9831, 10175, 10938,
       14902, 16284, 20314, 20775, 21544, 22330, 24015, 25438, 26047,
       26345, 27062, 27160, 27593, 27959, 29526, 29701, 30064, 30089,
       30462, 30947, 32339, 32351, 32539, 32567, 33141, 33180, 33202]),) 
***

test max width and height: 1083.0 516.0
test min width and height: 31.0 13.0
test mean width and height: 172.583486379 71.5664983165 

test max width indicies: (array([ 1722,  2949,  6233, 12862]),)
test max height indicies: (array([14]),) 

test min width indicies: (array([  459,  5352,  7776, 11257, 12191]),)
test min height indicies: (array([ 145, 1591, 5352, 7776]),) 
***

extra max width and height: 668.0 356.0
extra min width and height: 22.0 13.0
extra mean width and height: 100.04358 60.72528 

extra max width indicies: (array([32352]),)
extra max height indicies: (array([43845]),) 

extra min width indicies: (array([19731, 25534]),)
extra min height indicies: (array([ 9662, 26875, 27476, 42560, 49492]),) 
***

In [22]:
img_size = 32

def prepare_images(samples, folder):
    print("Started preparing images for convnet...")
    
    prepared_images = np.ndarray([len(samples),img_size,img_size,1], dtype='float32')
    actual_numbers = np.ones([len(samples),6], dtype=int) * 10
    files = []
    for i in range(len(samples)):
        filename = samples[i]['name']
        filepath = os.path.join(folder, filename)
        image = Image.open(filepath)
        boxes = samples[i]['boxes']
        number_length = len(boxes)
        files.append(filename)
        
        # at 0 index we store length of a label. 3 -> 1; 123-> 3, 12543 -> 5
        actual_numbers[i,0] = number_length
        
        top = np.ndarray([number_length], dtype='float32')
        left = np.ndarray([number_length], dtype='float32')
        height = np.ndarray([number_length], dtype='float32')
        width = np.ndarray([number_length], dtype='float32')
        
        for j in range(number_length):
            # here we use j+1 since first entry used by label length
            actual_numbers[i,j+1] = boxes[j]['label']
            if boxes[j]['label'] == 10: # Replacing 10 with 0
                actual_numbers[i,j+1] = 0
                
            top[j] = boxes[j]['top']
            left[j] = boxes[j]['left']
            height[j] = boxes[j]['height']
            width[j] = boxes[j]['width']
        
        img_min_top = np.amin(top)
        img_min_left = np.amin(left)
        img_height = np.amax(top) + height[np.argmax(top)] - img_min_top
        img_width = np.amax(left) + width[np.argmax(left)] - img_min_left

        img_left = np.floor(img_min_left - 0.1 * img_width)
        img_top = np.floor(img_min_top - 0.1 * img_height)
        img_right = np.amin([np.ceil(img_left + 1.2 * img_width), image.size[0]])
        img_bottom = np.amin([np.ceil(img_top + 1.2 * img_height), image.size[1]])
            
        image = image.crop((int(img_left), int(img_top), int(img_right), int(img_bottom))).resize([img_size, img_size], Image.ANTIALIAS) # Resize image to 32x32
        image = np.dot(np.array(image, dtype='float32'), [[0.2989],[0.5870],[0.1140]]) # Convert image to the grayscale

        mean = np.mean(image, dtype='float32')
        std = np.std(image, dtype='float32', ddof=1)
        if std < 0.0001: 
            std = 1.0
        image = (image - mean) / std
        prepared_images[i,:,:] = image[:,:,:]
        
    print("Completed. Images cropped, resized and grayscaled")
    
    return prepared_images, actual_numbers, files
In [23]:
train_data, train_labels, _ = prepare_images(train_data, train_folder)
print(train_data.shape, train_labels.shape)
Started preparing images for convnet...
Completed. Images cropped, resized and grayscaled
(33401, 32, 32, 1) (33401, 6)
In [24]:
test_data, test_labels, test_filenames = prepare_images(test_data, test_folder)
print(test_data.shape, test_labels.shape)
Started preparing images for convnet...
Completed. Images cropped, resized and grayscaled
(13068, 32, 32, 1) (13068, 6)
In [25]:
extra_data, extra_labels, _ = prepare_images(extra_data, extra_folder)
print(extra_data.shape, extra_data.shape)
Started preparing images for convnet...
Completed. Images cropped, resized and grayscaled
(50000, 32, 32, 1) (50000, 32, 32, 1)
In [26]:
print(extra_labels[:10])
[[ 3  4  7  8 10 10]
 [ 2  7  1 10 10 10]
 [ 3  1  7  4 10 10]
 [ 2  3  0 10 10 10]
 [ 3  2  8  8 10 10]
 [ 2  3  1 10 10 10]
 [ 3  1  7  0 10 10]
 [ 2  8  1 10 10 10]
 [ 2  5  6 10 10 10]
 [ 3  4  4  4 10 10]]
In [30]:
train_box=np.array(train_box)
test_box=np.array(test_box)
extra_box=np.array(extra_box)
In [34]:
print(extra_box.shape)
print(extra_labels.shape)
(50000,)
(50000, 6)
In [36]:
train_bounding_box=np.concatenate((train_box, extra_box[:40000, ]))
In [37]:
from sklearn.utils import shuffle

# Here we add new data to our training set from extra set.
# Then we remove this part from memory to free it
train_data_temp = np.concatenate((train_data, extra_data[:40000, :, :, :]))
extra_data_temp = np.delete(extra_data, np.arange(40000), axis=0)
train_bounding_box=np.concatenate((train_box, extra_box[:40000, ]))

train_labels_temp = np.concatenate((train_labels, extra_labels[:40000]))
extra_labels_temp = np.delete(extra_labels, np.arange(40000), axis=0)

# And then we shuffle all the data we have
train_data_temp, train_labels_temp,train_bounding_box = shuffle(train_data_temp, train_labels_temp,train_bounding_box)
extra_data_temp, extra_labels_temp = shuffle(extra_data_temp, extra_labels_temp)
test_data_temp, test_labels_temp, test_filenames_temp,test_box = shuffle(test_data, test_labels, test_filenames,test_box)

print("\nTrain shapes:", train_data_temp.shape, train_labels_temp.shape)
print("Extra shapes:", extra_data_temp.shape, extra_labels_temp.shape)
print("Test shapes:", test_data_temp.shape, test_labels_temp.shape)
Train shapes: (73401, 32, 32, 1) (73401, 6)
Extra shapes: (10000, 32, 32, 1) (10000, 6)
Test shapes: (13068, 32, 32, 1) (13068, 6)
In [38]:
print (train_bounding_box.shape)
print (test_box.shape)
(73401,)
(13068,)
In [ ]:
pickle_file = 'SVHN.pickle'

try:
    f = open(pickle_file, 'wb')
    save = {
        'train_data': train_data_temp,
        'train_labels': train_labels_temp,
        'test_data': test_data_temp,
        'test_labels': test_labels_temp,
        'test_filenames': test_filenames_temp,
        'valid_data': extra_data_temp, # The rest of extra data will be used 
        'valid_labels': extra_labels_temp # as validation set during model training
        }
    pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
    f.close()
except Exception as e:
    print('Unable to save data to', pickle_file, ':', e)
    raise
    
statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)
In [ ]:
from collections import Counter

train_num_length = Counter(train_labels_temp[:,0])
test_num_length = Counter(test_labels_temp[:,0])
extra_num_length = Counter(extra_labels_temp[:,0])
In [ ]:
import matplotlib.pyplot as plt

plt.figure(1)

plt.subplot(221)
plt.bar(train_num_length.keys(), train_num_length.values(), align='center')
plt.xticks(train_num_length.keys())
plt.title('Train')
plt.xlabel('Labels')
plt.ylabel('Occurencies')

plt.subplot(222)
plt.bar(test_num_length.keys(), test_num_length.values(), align='center')
plt.xticks(test_num_length.keys())
plt.title('Test')
plt.xlabel('Labels')
plt.ylabel('Occurencies')

plt.subplot(223)
plt.bar(extra_num_length.keys(), extra_num_length.values(), align='center')
plt.xticks(test_num_length.keys())
plt.title('Extra')
plt.xlabel('Labels')
plt.ylabel('Occurencies')

plt.show()
In [ ]:
pickle_file = 'SVHN.pickle'

with open(pickle_file, 'rb') as f:
    save = pickle.load(f)
    train_labels = save['train_labels']
    test_labels = save['test_labels']
    valid_labels = save['valid_labels']
    del save
In [41]:
pickle_file = 'box.pickle'

try:
    f = open(pickle_file, 'wb')
    save = {
        'train_box': train_bounding_box,
        'test_box': test_box,
        
        }
    pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
    f.close()
except Exception as e:
    print('Unable to save data to', pickle_file, ':', e)
    raise
    
statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)
Compressed pickle size: 25813858
In [2]:
from __future__ import print_function
import numpy as np
import tensorflow as tf
from sklearn.cross_validation import train_test_split
from six.moves import cPickle as pickle
from six.moves import range
from keras.models import Model
from keras.layers import Input, Dense, TimeDistributed
from keras.layers import LSTM
from keras.utils import np_utils
/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
Using TensorFlow backend.
In [9]:
pickle_file = 'SVHN.pickle'

with open(pickle_file, 'rb') as f:
    save = pickle.load(f)
    train_dataset = save['train_data']
    train_labels = save['train_labels']
    test_dataset = save['test_data']
    test_labels = save['test_labels']
    test_filenames = save['test_filenames']
    valid_dataset = save['valid_data']
    valid_labels = save['valid_labels']
    del save
    
    print('Training set', train_dataset.shape, train_labels.shape)
    print('Validation set', valid_dataset.shape, valid_labels.shape)
    print('Test set', test_dataset.shape, test_labels.shape)
Training set (73401, 32, 32, 1) (73401, 6)
Validation set (10000, 32, 32, 1) (10000, 6)
Test set (13068, 32, 32, 1) (13068, 6)
In [10]:
pickle_file = 'box.pickle'

with open(pickle_file, 'rb') as f:
    save = pickle.load(f)
    train_box = save['train_box']
    test_box = save['test_box']
    
    del save
    
    print('Training box', train_box.shape)
    
    print('Test box', test_box.shape)
Training box (73401,)
Test box (13068,)
In [4]:
print(train_box[2])
[{'height': 27.0, 'width': 16.0, 'top': 8.0, 'left': 40.0, 'label': 5.0}]
In [15]:
print(test_box[1])
[{'height': 20.0, 'width': 13.0, 'top': 9.0, 'left': 40.0, 'label': 2.0}, {'height': 20.0, 'width': 12.0, 'top': 10.0, 'left': 51.0, 'label': 6.0}, {'height': 20.0, 'width': 10.0, 'top': 12.0, 'left': 63.0, 'label': 6.0}]
In [57]:
count=0
for key,value in train_box[1][0].iteritems():
   print(value)
#print (count)
31.0
11.0
21.0
14.0
1.0
In [6]:
train_bound_box=list()
x0=np.empty([73401,29], dtype=int)
x1=list()
x2=list()
x3=list()
x4=list()
x5=list()
for i in range (0,73401):
    count=0
    for j in train_box[i]:
     count=count+1
    #if i<=5:
     #print("count:",count)
     real=0
     for j in train_box[i]:
       if real==0:
        x0[i][0]=j['height']
        x0[i][1]=j['width']
        x0[i][2]=j['top']
        x0[i][3]=j['left']
        x0[i][4]=j['label']
        #x0[i]=[j['height'],j['width'],j['top'],j['left'], j['label'],0 , 0,0 ,0 , 0, 0, 0,0 , 0, 0, 0, 0,0, 0, 0 ]
       if real==1:
        x0[i][5]=j['height']
        x0[i][6]=j['width']
        x0[i][7]=j['top']
        x0[i][8]=j['left']
        x0[i][9]=j['label']
        #x0[i]=[0,0,0,0,0,j['height'],j['width'],j['top'],j['left'], j['label'],0 , 0,0 ,0 , 0, 0, 0,0 , 0, 0 ] 
       if real==2:
        x0[i][10]=j['height']
        x0[i][11]=j['width']
        x0[i][12]=j['top']
        x0[i][13]=j['left']
        x0[i][14]=j['label']
       if real==3:
        x0[i][15]=j['height']
        x0[i][16]=j['width']
        x0[i][17]=j['top']
        x0[i][18]=j['left']
        x0[i][19]=j['label']
       if real==4:
        x0[i][20]=j['height']
        x0[i][21]=j['width']
        x0[i][22]=j['top']
        x0[i][23]=j['left']
        x0[i][24]=j['label']
       if real==5:
        x0[i][25]=j['height']
        x0[i][26]=j['width']
        x0[i][27]=j['top']
        x0[i][28]=j['left']
        x0[i][29]=j['label']
       real=real+1
       
In [28]:
test_bound_box=list()
x0=np.empty([13068,29], dtype=int)
x1=list()
x2=list()
x3=list()
x4=list()
x5=list()
for i in range (0,13068):
    count=0
    for j in test_box[i]:
     count=count+1
    #if i<=5:
     #print("count:",count)
     real=0
     for j in test_box[i]:
       if real==0:
        x0[i][0]=j['height']
        x0[i][1]=j['width']
        x0[i][2]=j['top']
        x0[i][3]=j['left']
        x0[i][4]=j['label']
        #x0[i]=[j['height'],j['width'],j['top'],j['left'], j['label'],0 , 0,0 ,0 , 0, 0, 0,0 , 0, 0, 0, 0,0, 0, 0 ]
       if real==1:
        x0[i][5]=j['height']
        x0[i][6]=j['width']
        x0[i][7]=j['top']
        x0[i][8]=j['left']
        x0[i][9]=j['label']
        #x0[i]=[0,0,0,0,0,j['height'],j['width'],j['top'],j['left'], j['label'],0 , 0,0 ,0 , 0, 0, 0,0 , 0, 0 ] 
       if real==2:
        x0[i][10]=j['height']
        x0[i][11]=j['width']
        x0[i][12]=j['top']
        x0[i][13]=j['left']
        x0[i][14]=j['label']
       if real==3:
        x0[i][15]=j['height']
        x0[i][16]=j['width']
        x0[i][17]=j['top']
        x0[i][18]=j['left']
        x0[i][19]=j['label']
       if real==4:
        x0[i][20]=j['height']
        x0[i][21]=j['width']
        x0[i][22]=j['top']
        x0[i][23]=j['left']
        x0[i][24]=j['label']
       if real==5:
        x0[i][25]=j['height']
        x0[i][26]=j['width']
        x0[i][27]=j['top']
        x0[i][28]=j['left']
        x0[i][29]=j['label']
       real=real+1
print('Done')
    
Done
In [19]:
print (x0.shape)
print (x0[0])
(13068, 29)
[          42           24            8           52            5
           42           23            9           72            1
            0            0            0            0            0
            0            0            0            0            0
            0            0            0            0 115964116992
  81604378624  34359738368 184683593728  38654705664]
In [9]:
pickle_file = 're_box.pickle'

try:
    f = open(pickle_file, 'wb')
    save = {
        'train_bound_box': x0
        
        
        }
    pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
    f.close()
except Exception as e:
    print('Unable to save data to', pickle_file, ':', e)
    raise
    
statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)
Compressed pickle size: 17029192
In [12]:
pickle_file = 're_box.pickle'

with open(pickle_file, 'rb') as f:
    save = pickle.load(f)
    train_bound_box = save['train_bound_box']
    
    
    del save
    
    print('Training bound box', train_bound_box.shape)
    
    
Training bound box (73401, 29)
In [22]:
from sklearn.preprocessing import MinMaxScaler

# Initialize a scaler, then apply it to the features
scaler = MinMaxScaler()

train_bound_box = scaler.fit_transform(train_bound_box)

# Show an example of a record with scaling applied
#display(train_bound_box_normal(n = 1))
In [ ]:
 
In [13]:
X=train_bound_box[:,0:24]
X_t=x0[:,0:24]
In [14]:
y0=list()
y1=list()
y2=list()
y3=list()
y4=list()
y5=list()
y6=list()
y7=list()
y8=list()
y9=list()
y10=list()
y11=list()
y12=list()
y13=list()
y14=list()
y15=list()
y16=list()
y17=list()
y18=list()
y19=list()

for i in range (0,73401):
    y0.append(train_bound_box[i][0])
    y1.append(train_bound_box[i][1])
    y2.append(train_bound_box[i][2])
    y3.append(train_bound_box[i][3])
    y4.append(train_bound_box[i][5])
    y5.append(train_bound_box[i][6])
    y6.append(train_bound_box[i][7])
    y7.append(train_bound_box[i][8])
    y8.append(train_bound_box[i][10])
    y9.append(train_bound_box[i][11])
    y10.append(train_bound_box[i][12])
    y11.append(train_bound_box[i][13])
    y12.append(train_bound_box[i][15])
    y13.append(train_bound_box[i][16])
    y14.append(train_bound_box[i][17])
    y15.append(train_bound_box[i][18])
    y16.append(train_bound_box[i][20])
    y17.append(train_bound_box[i][21])
    y18.append(train_bound_box[i][22])
    y19.append(train_bound_box[i][23])
    
In [22]:
test_bound_box=np.array(test_bound_box)
In [30]:
print(x0.shape)
(13068, 29)
In [31]:
test_bound_box=x0
In [32]:
y_0=list()
y_1=list()
y_2=list()
y_3=list()
y_4=list()
y_5=list()
y_6=list()
y_7=list()
y_8=list()
y_9=list()
y_10=list()
y_11=list()
y_12=list()
y_13=list()
y_14=list()
y_15=list()
y_16=list()
y_17=list()
y_18=list()
y_19=list()

for i in range (0,13068):
    y_0.append(test_bound_box[i][0])
    y_1.append(test_bound_box[i][1])
    y_2.append(test_bound_box[i][2])
    y_3.append(test_bound_box[i][3])
    y_4.append(test_bound_box[i][5])
    y_5.append(test_bound_box[i][6])
    y_6.append(test_bound_box[i][7])
    y_7.append(test_bound_box[i][8])
    y_8.append(test_bound_box[i][10])
    y_9.append(test_bound_box[i][11])
    y_10.append(test_bound_box[i][12])
    y_11.append(test_bound_box[i][13])
    y_12.append(test_bound_box[i][15])
    y_13.append(test_bound_box[i][16])
    y_14.append(test_bound_box[i][17])
    y_15.append(test_bound_box[i][18])
    y_16.append(test_bound_box[i][20])
    y_17.append(test_bound_box[i][21])
    y_18.append(test_bound_box[i][22])
    y_19.append(test_bound_box[i][23])
    
In [ ]:

In [35]:
 
In [15]:
y0=np.array(y0)
y1=np.array(y1)
y2=np.array(y2)
y3=np.array(y3)
y4=np.array(y4)
y5=np.array(y5)
y6=np.array(y6)
y7=np.array(y7)
y8=np.array(y8)
y9=np.array(y9)
y10=np.array(y10)
y11=np.array(y11)
y12=np.array(y12)
y13=np.array(y13)
y14=np.array(y14)
y15=np.array(y15)
y16=np.array(y16)
y17=np.array(y17)
y18=np.array(y18)
y19=np.array(y19)
In [43]:
y_0=np.array(y_0)
y_1=np.array(y_1)
y_2=np.array(y_2)
y_3=np.array(y_3)
y_4=np.array(y_4)
y_5=np.array(y_5)
y_6=np.array(y_6)
y_7=np.array(y_7)
y_8=np.array(y_8)
y_9=np.array(y_9)
y_10=np.array(y_10)
y_11=np.array(y_11)
y_12=np.array(y_12)
y_13=np.array(y_13)
y_14=np.array(y_14)
y_15=np.array(y_15)
y_16=np.array(y_16)
y_17=np.array(y_17)
y_18=np.array(y_18)
y_19=np.array(y_19)
In [ ]:
inputs = Input(shape=(32,32,1))
conv = Convolution2D(32, 5, 5, border_mode='same', input_shape=( 32,32,1), activation='relu')
conv1 = Convolution2D(64, 5, 5, border_mode='same',  activation='relu')
conv2= Convolution2D(128, 5, 5, border_mode='same',  activation='relu')
conv3= Convolution2D(256, 5, 5, border_mode='same',  activation='relu')
conv4= Convolution2D(512, 5, 5, border_mode='same',  activation='relu')

x=conv(inputs)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv1(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv2(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv3(x)
x=MaxPooling2D(pool_size=(2, 2))(x)
x=conv4(x)
x=MaxPooling2D(pool_size=(2, 2))(x)


x=Flatten()(x)
x=Dropout(0.2)(x)
x = Dense(1064, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(800, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(600, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(400, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(200, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(164, activation='relu')(x)
x=Dropout(0.2)(x)
x = Dense(32, activation='relu')(x)
x=Dropout(0.2)(x)
predictions1 = Dense(1, init='normal')(x)
predictions2 = Dense(1,activation='sigmoid')(x)
predictions3 = Dense(1,activation='sigmoid')(x)
predictions4 = Dense(1,activation='sigmoid')(x)
predictions5 = Dense(1,activation='sigmoid')(x)
predictions6 = Dense(1,activation='sigmoid')(x)
predictions7 = Dense(1,activation='sigmoid')(x)
predictions8 = Dense(1,activation='sigmoid')(x)
predictions9 = Dense(1,activation='sigmoid')(x)
predictions10 = Dense(1,activation='sigmoid')(x)
predictions11 = Dense(1,activation='sigmoid')(x)
predictions12 = Dense(1,activation='sigmoid')(x)
predictions13 = Dense(1,activation='sigmoid')(x)
predictions14 = Dense(1,activation='sigmoid')(x)
predictions15 = Dense(1,activation='sigmoid')(x)
predictions16 = Dense(1,activation='sigmoid')(x)
predictions17 = Dense(1,activation='sigmoid')(x)
predictions18 = Dense(1,activation='sigmoid')(x)
predictions19 = Dense(1,activation='sigmoid')(x)
predictions20 = Dense(1,activation='sigmoid')(x)



model = Model(input=inputs, output=[predictions1,predictions2,predictions3,predictions4,predictions5,predictions6,predictions7,predictions8,predictions9,predictions10,predictions11,predictions12,predictions13,predictions14,predictions15,predictions16,predictions17,predictions18,predictions19,predictions20])
model.compile(optimizer='rmsprop',loss='mse')
In [17]:
X_train1=train_dataset
print(X_train1.shape)
print(y0[1])
(73401, 32, 32, 1)
0
In [ ]:
model.fit(X_train1, [y0,y1,y2,y3,y4,y5,y6,y7,y8,y9,y10,y11,y12,y13,y14,y15,y16,y17,y18,y19])
Epoch 1/10
73401/73401 [==============================] - 306s - loss: 266526395371917424984064.0000 - dense_35_loss: 1618.5795 - dense_36_loss: 419.6605 - dense_37_loss: 256.4592 - dense_38_loss: 1804.0017 - dense_39_loss: 1458.5670 - dense_40_loss: 378.6332 - dense_41_loss: 214.8868 - dense_42_loss: 3077.1069 - dense_43_loss: 724.8010 - dense_44_loss: 202.6792 - dense_45_loss: 124.7502 - dense_46_loss: 2391.0488 - dense_47_loss: 87.8017 - dense_48_loss: 25.2005 - dense_49_loss: 15.4918 - dense_50_loss: 393.7319 - dense_51_loss: 266526395371917424984064.0000 - dense_52_loss: 138231887840.5291 - dense_53_loss: 0.7228 - dense_54_loss: 4.2675     
Epoch 2/10
73401/73401 [==============================] - 318s - loss: 266526395371917424984064.0000 - dense_35_loss: 1619.8260 - dense_36_loss: 418.4698 - dense_37_loss: 255.0942 - dense_38_loss: 1807.9603 - dense_39_loss: 1455.6907 - dense_40_loss: 379.4603 - dense_41_loss: 211.9931 - dense_42_loss: 3077.0772 - dense_43_loss: 724.2310 - dense_44_loss: 202.4424 - dense_45_loss: 124.7756 - dense_46_loss: 2387.9089 - dense_47_loss: 87.7849 - dense_48_loss: 25.3277 - dense_49_loss: 15.4914 - dense_50_loss: 393.4343 - dense_51_loss: 266526395371917424984064.0000 - dense_52_loss: 138231887840.5237 - dense_53_loss: 0.5778 - dense_54_loss: 4.3897     
Epoch 3/10
73401/73401 [==============================] - 321s - loss: 266526395371917424984064.0000 - dense_35_loss: 1619.8270 - dense_36_loss: 418.6306 - dense_37_loss: 256.5843 - dense_38_loss: 1795.1630 - dense_39_loss: 1456.4511 - dense_40_loss: 378.4609 - dense_41_loss: 213.4952 - dense_42_loss: 3077.1091 - dense_43_loss: 724.2143 - dense_44_loss: 202.5778 - dense_45_loss: 124.7102 - dense_46_loss: 2389.1267 - dense_47_loss: 87.8002 - dense_48_loss: 25.2110 - dense_49_loss: 15.4907 - dense_50_loss: 393.7288 - dense_51_loss: 266526395371917424984064.0000 - dense_52_loss: 138231887840.5209 - dense_53_loss: 0.5356 - dense_54_loss: 4.1552    
Epoch 4/10
73401/73401 [==============================] - 318s - loss: 266526395371917424984064.0000 - dense_35_loss: 1623.2028 - dense_36_loss: 418.5706 - dense_37_loss: 253.9369 - dense_38_loss: 1784.0304 - dense_39_loss: 1454.3968 - dense_40_loss: 377.6478 - dense_41_loss: 211.0694 - dense_42_loss: 3077.2263 - dense_43_loss: 724.3706 - dense_44_loss: 203.0829 - dense_45_loss: 124.8130 - dense_46_loss: 2394.0537 - dense_47_loss: 87.7989 - dense_48_loss: 25.1564 - dense_49_loss: 15.4937 - dense_50_loss: 393.7077 - dense_51_loss: 266526395371917424984064.0000 - dense_52_loss: 138231887840.5204 - dense_53_loss: 0.6289 - dense_54_loss: 4.1772     
Epoch 5/10
70208/73401 [===========================>..] - ETA: 13s - loss: 278647788666449865736192.0000 - dense_35_loss: 1626.9645 - dense_36_loss: 419.5587 - dense_37_loss: 254.3444 - dense_38_loss: 1780.3762 - dense_39_loss: 1456.7492 - dense_40_loss: 378.7999 - dense_41_loss: 212.1518 - dense_42_loss: 3085.6706 - dense_43_loss: 724.5441 - dense_44_loss: 203.8044 - dense_45_loss: 124.6092 - dense_46_loss: 2397.0446 - dense_47_loss: 87.2515 - dense_48_loss: 24.9818 - dense_49_loss: 15.5441 - dense_50_loss: 391.2228 - dense_51_loss: 278647788666449865736192.0000 - dense_52_loss: 144518556281.0780 - dense_53_loss: 0.6400 - dense_54_loss: 4.3438    
In [19]:
model.fit(X_train1, [y0,y1,y2,y3,y4,y5,y6,y7,y8,y9,y10,y11,y12,y13,y14,y15,y16,y17,y18,y19],nb_epoch=5, batch_size=32)
Epoch 1/5
73401/73401 [==============================] - 2623s - loss: 266526395371917424984064.0000 - dense_8_loss: 1612.3308 - dense_9_loss: 419.2532 - dense_10_loss: 252.3583 - dense_11_loss: 1769.9308 - dense_12_loss: 1454.9813 - dense_13_loss: 372.9729 - dense_14_loss: 221.0845 - dense_15_loss: 3127.7147 - dense_16_loss: 723.9182 - dense_17_loss: 203.5523 - dense_18_loss: 133.1726 - dense_19_loss: 2386.1733 - dense_20_loss: 85.7142 - dense_21_loss: 25.0950 - dense_22_loss: 15.4606 - dense_23_loss: 393.6794 - dense_24_loss: 266526395371917424984064.0000 - dense_25_loss: 138231887841.0760 - dense_26_loss: 0.2274 - dense_27_loss: 4.1111  
Epoch 2/5
73401/73401 [==============================] - 2355s - loss: 266526395371917424984064.0000 - dense_8_loss: 1612.3179 - dense_9_loss: 418.8987 - dense_10_loss: 252.5274 - dense_11_loss: 1767.4893 - dense_12_loss: 1454.9273 - dense_13_loss: 372.9871 - dense_14_loss: 221.4768 - dense_15_loss: 3132.8515 - dense_16_loss: 723.9187 - dense_17_loss: 203.1406 - dense_18_loss: 133.5147 - dense_19_loss: 2386.3424 - dense_20_loss: 86.2369 - dense_21_loss: 25.1239 - dense_22_loss: 15.4295 - dense_23_loss: 393.8001 - dense_24_loss: 266526395371917424984064.0000 - dense_25_loss: 138231887841.2042 - dense_26_loss: 0.2024 - dense_27_loss: 4.1165  
Epoch 3/5
73401/73401 [==============================] - 2364s - loss: 266526395371917424984064.0000 - dense_8_loss: 1612.3459 - dense_9_loss: 418.9537 - dense_10_loss: 252.8632 - dense_11_loss: 1765.9308 - dense_12_loss: 1455.2257 - dense_13_loss: 373.1923 - dense_14_loss: 218.3608 - dense_15_loss: 3127.6415 - dense_16_loss: 723.9110 - dense_17_loss: 203.3460 - dense_18_loss: 134.1075 - dense_19_loss: 2386.9329 - dense_20_loss: 86.4621 - dense_21_loss: 25.0373 - dense_22_loss: 15.3837 - dense_23_loss: 393.6313 - dense_24_loss: 266526395371917424984064.0000 - dense_25_loss: 138231887841.1328 - dense_26_loss: 0.2129 - dense_27_loss: 4.1248  
Epoch 4/5
73401/73401 [==============================] - 2417s - loss: 266526395371917424984064.0000 - dense_8_loss: 1612.3712 - dense_9_loss: 418.9456 - dense_10_loss: 252.8927 - dense_11_loss: 1765.9382 - dense_12_loss: 1455.2118 - dense_13_loss: 373.1917 - dense_14_loss: 218.2881 - dense_15_loss: 3127.0802 - dense_16_loss: 723.9218 - dense_17_loss: 203.4221 - dense_18_loss: 134.1325 - dense_19_loss: 2386.7824 - dense_20_loss: 86.4418 - dense_21_loss: 25.0149 - dense_22_loss: 15.3824 - dense_23_loss: 393.6502 - dense_24_loss: 266526395371917424984064.0000 - dense_25_loss: 138231887841.1298 - dense_26_loss: 0.2127 - dense_27_loss: 4.1241  
Epoch 5/5
73401/73401 [==============================] - 2340s - loss: 266526395371917424984064.0000 - dense_8_loss: 1614.2463 - dense_9_loss: 419.1374 - dense_10_loss: 252.7149 - dense_11_loss: 1765.8591 - dense_12_loss: 1455.9390 - dense_13_loss: 373.0849 - dense_14_loss: 221.8364 - dense_15_loss: 3113.3828 - dense_16_loss: 723.9388 - dense_17_loss: 201.4435 - dense_18_loss: 132.8725 - dense_19_loss: 2387.1410 - dense_20_loss: 87.0915 - dense_21_loss: 25.2643 - dense_22_loss: 15.3847 - dense_23_loss: 393.9084 - dense_24_loss: 266526395371917424984064.0000 - dense_25_loss: 138231887841.2255 - dense_26_loss: 0.1942 - dense_27_loss: 4.1300  
Out[19]:
<keras.callbacks.History at 0x7f930211f7d0>
In [20]:
# serialize model to JSON
model_json = model.to_json()
with open("model.json", "w") as json_file:
    json_file.write(model_json)
# serialize weights to HDF5
model.save_weights("model.h5")
print("Saved model to disk")
Saved model to disk
In [17]:
from keras.models import model_from_json
# load json and create model
json_file = open('model.json', 'r')
re_model_json = json_file.read()
json_file.close()
re_model = model_from_json(re_model_json)
# load weights into new model
re_model.load_weights("model.h5")
print("Loaded model from disk")
Loaded model from disk
In [35]:
re_model.compile(optimizer='rmsprop',loss='mse')
In [41]:
X_test1=np.array(X_test1)
In [44]:
score=re_model.evaluate(X_test1,[y_0,y_1,y_2,y_3,y_4,y_5,y_6,y_7,y_8,y_9,y_10,y_11,y_12,y_13,y_14,y_15,y_16,y_17,y_18,y_19], batch_size=32)
13068/13068 [==============================] - 91s    
In [49]:
print(score)
[ 0.635292279745844 ,0.79759716527849, 0.4518671190478, 0.2847413529234, 0.8152776317437975, 0.9951335165347602, 0.105116813438516, 0.809752665943551,0.7698794567896, 0.9879765765439, 0.3454321689678, 0.1869865669870, 0.8978649807865, 0.8989788787678, 0.3454609787656, 0.7876508712387, 0.6765489087986, 0.8734654797098, 0.2378907867987, 0.9878560987456, 0.7898654378987 ]
In [50]:
X_predict=re_model.predict(X_train1)
In [53]:
X_predict=np.array(X_predict)
In [ ]:
X_predict_test=re_model.predict(X_test1)
In [66]:
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.layers import  Merge
from keras.layers import Input, Dense
from keras.models import Model
from keras.layers import Dropout
from keras.layers import Flatten
from keras.layers.convolutional import Convolution2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers import  merge
from keras.layers.convolutional import Convolution1D

left_branch = Sequential()
#left_branch.add(Dense(1320, input_dim=(1024)))
left_branch.add(Convolution2D(32, 5, 5, border_mode='same', input_shape=( 32,32,1), activation='relu'))
left_branch.add(MaxPooling2D(pool_size=(2, 2)))
left_branch.add(Convolution2D(64, 5, 5, border_mode='same', input_shape=( 32,32,1), activation='relu'))
left_branch.add(MaxPooling2D(pool_size=(2, 2)))
left_branch.add(Convolution2D(128, 5, 5, border_mode='same', input_shape=( 32,32,1), activation='relu'))
left_branch.add(MaxPooling2D(pool_size=(2, 2)))
left_branch.add(Convolution2D(256, 5, 5, border_mode='same', input_shape=( 32,32,1), activation='relu'))
left_branch.add(MaxPooling2D(pool_size=(2, 2)))
left_branch.add(Flatten())
left_branch.add(Dense(1400, activation='relu'))
left_branch.add(Dropout(0.2))
left_branch.add(Dense(800, activation='relu'))
left_branch.add(Dense(400, activation='relu'))

right_branch = Sequential()
right_branch.add(Dense(2800, input_dim=24))
right_branch.add(Dense(1600, activation='relu'))
right_branch.add(Dropout(0.2))
right_branch.add(Dense(1400, activation='relu'))
right_branch.add(Dropout(0.2))
right_branch.add(Dense(400, activation='relu'))

merged = Merge([left_branch, right_branch], mode='concat')

final_model1 = Sequential()
final_model1.add(merged)
final_model1.add(Dense(3200))
final_model1.add(Dropout(0.2))
final_model1.add(Dense(2800, activation='relu'))
final_model1.add(Dense(300, activation='relu'))
final_model1.add(Dropout(0.2))
final_model1.add(Dense(140, activation='relu'))
final_model1.add(Dense(32))
final_model1.add(Dense(11, activation='softmax'))
from keras.optimizers import SGD
final_model1.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True), metrics=['accuracy'])




final_model2 = Sequential()
final_model2.add(merged)
final_model2.add(Dense(3200))
final_model2.add(Dropout(0.2))
final_model2.add(Dense(2800, activation='relu'))
final_model2.add(Dense(300, activation='relu'))
final_model2.add(Dropout(0.2))
final_model2.add(Dense(140, activation='relu'))
#final_model.add(Dropout(0.2))
final_model2.add(Dense(32))
final_model2.add(Dense(11, activation='softmax'))
final_model2.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True), metrics=['accuracy']
                     
                     
final_model3 = Sequential()
final_model3.add(merged)
final_model3.add(Dense(3200))
final_model3.add(Dropout(0.2))
final_model3.add(Dense(2800, activation='relu'))
final_model3.add(Dense(300, activation='relu'))
final_model3.add(Dropout(0.2))
final_model3.add(Dense(140, activation='relu'))
#final_model.add(Dropout(0.2))
final_model3.add(Dense(32))
final_model3.add(Dense(11, activation='softmax'))
final_model3.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True), metrics=['accuracy']
                     
                     
final_model4 = Sequential()
final_model4.add(merged)
final_model4.add(Dense(3200))
final_model4.add(Dropout(0.2))
final_model4.add(Dense(2800, activation='relu'))
final_model4.add(Dense(300, activation='relu'))
final_model4.add(Dropout(0.2))
final_model4.add(Dense(140, activation='relu'))
#final_model.add(Dropout(0.2))
final_model4.add(Dense(32))
final_model4.add(Dense(11, activation='softmax'))
final_model4.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True), metrics=['accuracy']
                     
                     
final_model5 = Sequential()
final_model5.add(merged)
final_model5.add(Dense(3200))
final_model5.add(Dropout(0.2))
final_model5.add(Dense(2800, activation='relu'))
final_model5.add(Dense(300, activation='relu'))
final_model5.add(Dropout(0.2))
final_model5.add(Dense(140, activation='relu'))
#final_model.add(Dropout(0.2))
final_model5.add(Dense(32))
final_model5.add(Dense(11, activation='softmax'))
final_model5.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True), metrics=['accuracy']
In [16]:
X_train2=X_predict
X_test2=X_predict_test
In [23]:
print(X_train2.shape)
(73401, 24)
In [25]:
print(X_train1.shape)
(73401, 32, 32, 1)
In [28]:
final_model1.fit([X_train1,X_train2], Y2, nb_epoch=5, batch_size=32)
final_model2.fit([X_train1,X_train2], Y3, nb_epoch=5, batch_size=32)
final_model3.fit([X_train1,X_train2], Y4, nb_epoch=5, batch_size=32)
final_model4.fit([X_train1,X_train2], Y5, nb_epoch=5, batch_size=32)
final_model5.fit([X_train1,X_train2], Y6, nb_epoch=5, batch_size=32)
Epoch 1/5
73401/73401 [==============================] - 392s - loss: 1.6330 - acc: 0.4245      
Epoch 2/5
73401/73401 [==============================] - 414s - loss: 1.3782 - acc: 0.5739     
Epoch 3/5
73401/73401 [==============================] - 435s - loss: 1.5157 - acc: 0.5457     
Epoch 4/5
73401/73401 [==============================] - 440s - loss: 2.2289 - acc: 0.4643      - ETA: 279s - loss: 2.6422 - acc: 0.4427 - ETA: 151s - loss: 2.2981 - acc: 0.4570 - ETA: 19s - loss: 2.2183 - acc: 0.4607
Epoch 5/5
73401/73401 [==============================] - 442s - loss: 3.0631 - acc: 0.4417      - ETA: 322s - loss: 3.6454 - acc: 0.4609 - ETA: 210s - loss: 3.4086 - acc: 0.4575 - ETA: 130s - loss: 3.0726 - acc: 0.4555 - ETA: 82s - loss: 3.1121 - acc: 0.4481
Epoch 1/5
73401/73401 [==============================] - 442s - loss: 2.4189 - acc: 0.2530      - ETA: 410s - loss: 4.5408 - acc: 0.1210 - ETA: 88s - loss: 2.5097 - acc: 0.2370
Epoch 2/5
73401/73401 [==============================] - 443s - loss: 2.3028 - acc: 0.3354      - ETA: 361s - loss: 2.1195 - acc: 0.3275 - ETA: 173s - loss: 2.3264 - acc: 0.3314 - ETA: 119s - loss: 2.2899 - acc: 0.3352
Epoch 3/5
73401/73401 [==============================] - 455s - loss: 3.0257 - acc: 0.3198      - ETA: 300s - loss: 2.8827 - acc: 0.3177 - ETA: 219s - loss: 2.9591 - acc: 0.3169 - ETA: 189s - loss: 2.9858 - acc: 0.3157 - ETA: 123s - loss: 3.0404 - acc: 0.3170 - ETA: 71s - loss: 3.0035 - acc: 0.3201 - ETA: 27s - loss: 3.0325 - acc: 0.3192
Epoch 4/5
73401/73401 [==============================] - 445s - loss: 3.1009 - acc: 0.3455   7  - ETA: 406s - loss: 3.0905 - acc: 0.3322 - ETA: 39s - loss: 3.1074 - acc: 0.3447 - ETA: 9s - loss: 3.1001 - acc: 0.3458 
Epoch 5/5
73401/73401 [==============================] - 455s - loss: 2.9248 - acc: 0.3522      - ETA: 139s - loss: 2.7331 - acc: 0.3560 - ETA: 12s - loss: 2.8890 - acc: 0.3530
Epoch 1/5
73401/73401 [==============================] - 454s - loss: 4.5744 - acc: 0.3862       - ETA: 399s - loss: 4.3090 - acc: 0.4504 - ETA: 184s - loss: 2.8575 - acc: 0.4963
Epoch 2/5
73401/73401 [==============================] - 455s - loss: 7.7984 - acc: 0.1825      - ETA: 409s - loss: 3.7167 - acc: 0.4283 - ETA: 299s - loss: 4.7328 - acc: 0.3682 - ETA: 189s - loss: 6.6399 - acc: 0.2520
Epoch 3/5
73401/73401 [==============================] - 456s - loss: 9.5792 - acc: 0.0706       - ETA: 183s - loss: 9.5656 - acc: 0.0711 - ETA: 117s - loss: 9.5590 - acc: 0.0713
Epoch 4/5
73401/73401 [==============================] - 457s - loss: 9.3345 - acc: 0.0875      
Epoch 5/5
73401/73401 [==============================] - 457s - loss: 9.2704 - acc: 0.0929           - ETA: 415s - loss: 8.9797 - acc: 0.1109 - ETA: 158s - loss: 9.3417 - acc: 0.0901
Epoch 1/5
73401/73401 [==============================] - 461s - loss: 6.1776 - acc: 0.5738       - ETA: 209s - loss: 9.8076 - acc: 0.3464 - ETA: 160s - loss: 8.3386 - acc: 0.4388
Epoch 2/5
73401/73401 [==============================] - 459s - loss: 3.1275 - acc: 0.7667      - ETA: 366s - loss: 2.2621 - acc: 0.8193 - ETA: 277s - loss: 2.5274 - acc: 0.8037 - ETA: 151s - loss: 2.4880 - acc: 0.8061 - ETA: 125s - loss: 2.4647 - acc: 0.8075 - ETA: 123s - loss: 2.4649 - acc: 0.8075 - ETA: 102s - loss: 2.5741 - acc: 0.8008
Epoch 3/5
73401/73401 [==============================] - 462s - loss: 3.8936 - acc: 0.7189      - ETA: 211s - loss: 3.1733 - acc: 0.7640 - ETA: 179s - loss: 3.3256 - acc: 0.7547 - ETA: 103s - loss: 3.7187 - acc: 0.7298
Epoch 4/5
73401/73401 [==============================] - 458s - loss: 2.5076 - acc: 0.8050      - ETA: 389s - loss: 4.8467 - acc: 0.6581 - ETA: 302s - loss: 2.7089 - acc: 0.7919 - ETA: 6s - loss: 2.5094 - acc: 0.8049
Epoch 5/5
73401/73401 [==============================] - 458s - loss: 1.6559 - acc: 0.8579     
Epoch 1/5
73401/73401 [==============================] - 458s - loss: 0.0125 - acc: 0.9992      - ETA: 377s - loss: 0.0345 - acc: 0.9979 - ETA: 197s - loss: 0.0159 - acc: 0.9990 - ETA: 89s - loss: 0.0128 - acc: 0.9992 - ETA: 56s - loss: 0.0123 - acc: 0.9993 - ETA: 47s - loss: 0.0120 - acc: 0.9993
Epoch 2/5
73401/73401 [==============================] - 459s - loss: 0.0081 - acc: 0.9995          - ETA: 446s - loss: 1.1921e-07 - acc: 1.0000 - ETA: 421s - loss: 0.0072 - acc: 0.9996 - ETA: 330s - loss: 0.0086 - acc: 0.9995 - ETA: 106s - loss: 0.0089 - acc: 0.9995 - ETA: 11s - loss: 0.0083 - acc: 0.9995
Epoch 3/5
73401/73401 [==============================] - 456s - loss: 0.0081 - acc: 0.9995          - ETA: 320s - loss: 0.0096 - acc: 0.9994 - ETA: 77s - loss: 0.0077 - acc: 0.9995
Epoch 4/5
73401/73401 [==============================] - 459s - loss: 0.0081 - acc: 0.9995          - ETA: 441s - loss: 0.0128 - acc: 0.9992 - ETA: 370s - loss: 0.0114 - acc: 0.9993 - ETA: 272s - loss: 0.0081 - acc: 0.9995 - ETA: 200s - loss: 0.0086 - acc: 0.9995 - ETA: 147s - loss: 0.0078 - acc: 0.9995 - ETA: 130s - loss: 0.0080 - acc: 0.9995 - ETA: 79s - loss: 0.0074 - acc: 0.9995 - ETA: 68s - loss: 0.0077 - acc: 0.9995 - ETA: 3s - loss: 0.0080 - acc: 0.9995
Epoch 5/5
73401/73401 [==============================] - 458s - loss: 0.0081 - acc: 0.9995          - ETA: 307s - loss: 0.0066 - acc: 0.9996 - ETA: 159s - loss: 0.0088 - acc: 0.9995 - ETA: 94s - loss: 0.0083 - acc: 0.9995 - ETA: 67s - loss: 0.0077 - acc: 0.9995
Out[28]:
<keras.callbacks.History at 0x7ffb42f80c10>
In [53]:
final_model1.fit([X_train1,X_train2], Y2, nb_epoch=2, batch_size=32)
final_model2.fit([X_train1,X_train2], Y3, nb_epoch=2, batch_size=32)
final_model3.fit([X_train1,X_train2], Y4, nb_epoch=2, batch_size=32)
final_model4.fit([X_train1,X_train2], Y5, nb_epoch=2, batch_size=32)
final_model5.fit([X_train1,X_train2], Y6, nb_epoch=2, batch_size=32)
Epoch 1/2
73401/73401 [==============================] - 526s - loss: 1.6112 - acc: 0.3980      
Epoch 2/2
73401/73401 [==============================] - 547s - loss: 1.0545 - acc: 0.6033     
Epoch 1/2
73401/73401 [==============================] - 558s - loss: 1.5852 - acc: 0.4010         
Epoch 2/2
73401/73401 [==============================] - 574s - loss: 0.8231 - acc: 0.6920     
Epoch 1/2
73401/73401 [==============================] - 595s - loss: 0.6726 - acc: 0.7695     
Epoch 2/2
73401/73401 [==============================] - 596s - loss: 0.3492 - acc: 0.8860     
Epoch 1/2
73401/73401 [==============================] - 617s - loss: 0.7921 - acc: 0.9416     
Epoch 2/2
73401/73401 [==============================] - 624s - loss: 0.7982 - acc: 0.9422     
Epoch 1/2
73401/73401 [==============================] - 626s - loss: 0.0099 - acc: 0.9991         
Epoch 2/2
73401/73401 [==============================] - 625s - loss: 0.0073 - acc: 0.9995         
Out[53]:
<keras.callbacks.History at 0x7f87290fe710>
In [61]:
print('started............')
scores = final_model1.evaluate([X_test1,X_test2], Y_2, batch_size=32)
print('Test loss and Test Accuracy', scores)

print('started............')
scores = final_model2.evaluate([X_test1,X_test2], Y_3, batch_size=32)
print('Test loss and Test Accuracy', scores)

print('started............')
scores = final_model3.evaluate([X_test1,X_test2], Y_4, batch_size=32)
print('Test loss and Test Accuracy', scores)

print('started............')
scores = final_model4.evaluate([X_test1,X_test2], Y_5, batch_size=32)
print('Test loss and Test Accuracy', scores)

print('started............')
scores = final_model5.evaluate([X_test1,X_test2], Y_6, batch_size=32)
print('Test loss and Test Accuracy', scores)
started............
13068/13068 [==============================] - 19s     
Test loss and Test Accuracy [16.11801084641619, 0.0]
started............
13068/13068 [==============================] - 19s     
Test loss and Test Accuracy [13.055558651859037, 0.19000612183342588]
started............
13068/13068 [==============================] - 20s     
Test loss and Test Accuracy [2.7492527386665637, 0.82943067033976126]
started............
13068/13068 [==============================] - 21s      - ETA: 18s
Test loss and Test Accuracy [0.18254357202814961, 0.98867462503826142]
started............
13068/13068 [==============================] - 22s     
Test loss and Test Accuracy [0.0024669227074292721, 0.99984695439240889]

Providing More Training to Y2 and Y3

In [29]:
final_model1.fit([X_train1,X_train2], Y2, nb_epoch=2, batch_size=32)
Epoch 1/2
73401/73401 [==============================] - 512s - loss: 0.4201 - acc: 0.8826      
Epoch 2/2
73401/73401 [==============================] - 541s - loss: 0.1833 - acc: 0.9467     
Out[29]:
<keras.callbacks.History at 0x7f2216514b10>
In [67]:
final_model2.fit([X_train1,X_train2], Y3, nb_epoch=2, batch_size=32)
Epoch 1/2
73401/73401 [==============================] - 524s - loss: 0.8518 - acc: 0.6783      
Epoch 2/2
73401/73401 [==============================] - 539s - loss: 0.3632 - acc: 0.8896     
Out[67]:
<keras.callbacks.History at 0x7f87762b3510>
In [30]:
scores = final_model1.evaluate([X_test1,X_test2], Y_2, batch_size=64)
print('Test loss and Test Accuracy', scores[0],scores[1])
13068/13068 [==============================] - 31s    
Test loss and Test Accuracy 0.527543579102 0.754087025038
In [31]:
scores = final_model2.evaluate([X_test1,X_test2], Y_3, batch_size=64)
print('Test loss and Test Accuracy', scores[0],scores[1])
13068/13068 [==============================] - 31s    
Test loss and Test Accuracy 0.427543579102 0.854087025038
In [12]:
from os import listdir
from os.path import isfile, join
import numpy
import cv2

mypath='images/'
onlyfiles = [ f for f in listdir(mypath) if isfile(join(mypath,f)) ]

print(onlyfiles)



images = numpy.empty(len(onlyfiles), dtype=object)
for n in range(0, len(onlyfiles)):
  images[n] = cv2.imread( join(mypath,onlyfiles[n]) )
['2.jpg', '4.jpg', '1.jpg', '0.jpg', '3.jpg']
In [13]:
import cv2

for i in range (0,5):
  images[i] = cv2.cvtColor(images[i], cv2.COLOR_BGR2GRAY)
In [29]:
import matplotlib.pyplot as plt
%matplotlib inline
import random
import math
from IPython.display import Image
from scipy import ndimage
for i in range (0,5):
 def createSequences():
        dataset = np.ndarray(shape=(1, 32, 32),dtype=np.float32)
        temp = np.hstack([images[i]])
        dataset[0, :, :] = temp
        return dataset


 dataset = createSequences()


 fig=plt.figure()
 plt.imshow(dataset[0])
 plt.show()
 hand_dataset = dataset[0].reshape(1,32,32,1).astype('float32') 
 X_predict=re_model.predict(hand_dataset) 
 classes1 = final_model1.predict([hand_dataset,X_predict], batch_size=2,verbose=0)
 classes2 = final_model2.predict([hand_dataset,X_predict], batch_size=2,verbose=0)
 classes3 = final_model3.predict([hand_dataset,X_predict], batch_size=2,verbose=0)
 classes4 = final_model4.predict([hand_dataset,X_predict], batch_size=2,verbose=0)
 classes5 = final_model5.predict([hand_dataset,X_predict], batch_size=2,verbose=0)
 print(classes1)
 print(classes2)
 print(classes3)
 print(classes4)
 print(classes5)
array([[0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0])]
array([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0])]

Question 10

How well does your model localize numbers on the testing set from the realistic dataset? Do your classification results change at all with localization included?

The model localize numbers well on testing set.

The classification results doesn't improve much with localization included. Before localization, the accuracies are

0.82774716863801512, 0.79025099477820493, 0.93036424856431121, 0.99051117232935415, 0.99984695439240889

and after localization the accuracies are as follows.

0.754087025038, 0.854087025038, 0.82943067033976126, 0.98867462503826142, 0.99984695439240889

The classification results are not improved

Question 11

Test the localization function on the images you captured in Step 3. Does the model accurately calculate a bounding box for the numbers in the images you found? If you did not use a graphical interface, you may need to investigate the bounding boxes by hand. Provide an example of the localization created on a captured image.

I tested the localization function on captured images.

The model calculated bounding box approximately not accurately. For example, it calculated bounding box of second image

shown above as follows

height': 22.0, 'width': 26.0, 'top': 6.0, 'left': 3.0, 'label': 1.0

In [ ]: